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Project  summary.  This  Small  Business  Technology  Transfer  Phase  I  project  advanced  the 
state  of  the  art  in  formal  modeling  and  engineering  of  complex  distributed  systems.  The  pursued 
research  and  development  approach  included:  (a)  modeling  language(s)  that  can  be  used  to  repre¬ 
sent  complex  distributed  systems  and  their  components,  theory  and  methodology  providing  sound 
mathematical  basis  for  modeling  systems  and  reasoning  about  their  properties,  (fo)  extensible  and 
scalable  analysis  tools  that  can  be  used  to  validate  correctness  and  performance  properties,  and  au¬ 
tomated  synthesis. tools  that  ean  produce  efficient  deployment  schemes  of  the  software  components 
in  target  networks  subject  to  specified  constraints. 

In  prior  research  we  developed  an  expressive  modeling  language,  called  Tempo,  that  can  be 
used  to  represent  complex  distributed  systems  and  their  components.  We  also  have  developed  the 
theory  and  methodology  providing  sound  mathematical  basis  for  modeling  systems  and  reasoning 
about  their  properties,  along  with  tools  that  can  be  used  to  validate  correctness  and  performance 
properties.  In  this  project  we  have  extended  the  methodology  to  incorporate  additional  means  for 
reasoning  about  probabilistic  and  hybrid  systems.  Based  on  these  developments,  we  built  an  ex¬ 
tended  integrated  development  environment,  called  Tempo,  for  modeling,  synthesis,  and  analysis  of 
distributed  systems  We  have  also  prototyped  a  methodology  that  can  be  used  to  generate  working 
code  from  system  models  and  yield  efficient  deployment  of  the  software  components  in  target  net¬ 
works.  The  ultimate  goal  of  this  work  is  to  develop  a  complete  and  comprehensive  formal  method¬ 
ology  based  in  sound  theory,  and  au  industrial-grade  extensible  integrated  software  engineering 
environment  for  the  implementors  of  modern  distributed  software  systems.  The  preliminary  release 
of  the  system  for  Linux,  Windows,  and  OSX-PPC  platforms  is  available  at  www.veromodo.com. 
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1  Introduction 

This  is  the  linal  technical  report  for  Phase  I  STTR  project  that  focused  on  extensions  to  the  TIOA 
(Timed  Input/Output  Automata)  language  and  framework  as  well  as  its  companion  system:  Tempo. 
Specifically,  it  describes  the  advancements  in  the  theory  of  TIOA  and  the  addition  of  a  language 
extension  and  software  tool  back-end  aimed  at  assisting  users  of  the  methodology  when  turning 
their  attention  to  the  implementation  aspects  of  their  distributed  system.  The  new  extension  makes 
it  possible  to  specify  the  characteristics  of  a  deployment  environment  at  the  TIOA  level  with  model 
annotations.  The  annotations  are  then  used  by  Tempo  to  derive  a  combinatorial  optimization  model 
that  produces  an  optimal  (with  respect  to  an  objective  specified  in  the  annotations)  deployment 
scenario.  The  back-end  tool  relies  on  state-of-the-art  combinatorial  optimization  tool  to  solve  the 
optimization  problem.  The  Tempo  tool-chain  now  offers  an  end-to-end  solution  starting  with  the 
specification  of  a  distributed  algorithm  to  its  optimal  deployment,  on  a  target  platform. 

1.1  The  problem  and  our  solution 

Challenges  in  developing  distributed  systems.  Developing  dependable  distributed  systems 
for  modern  computing  platforms  continues  to  be  challenging.  While  the  availability  of  distributed 
middleware  makes  feasible  the  construction  of  systems  that  rim  on  distributed  platforms,  ensuring 
that  the  resulting  systems  satisfy  specific  safety,  timing,  and  fault-tolerance  requirements  remains 
problematic.  The  middleware  services  used  for  constructing  distributed  software  are  specified  in¬ 
formally  and  without  precise  guarantees  of  efficiency,  timing,  scalability,  compositionality,  and 
fanlt-tole.rance.  Even  when  services  and  algorithms  are  specified  formally,  rigorous  reasoning  about 
the  specifications  is  often  left  out  of  the  development  process. 

As  contemporary  distributed  systems  continue  to  grow  in  complexity  and  sophistication  in  many 
domains,  these  systems  are  required  to  have  formally-specified  guarantees  of  safety,  performance, 
and  fault-tolerance.  Current  software-engineering  practice  limits  the  specification  of  such  require¬ 
ments  to  informal  descriptions.  When  formal  specifications  are  given,  they  are  typically  provided 
only  for  the  system  interfaces.  The  specification  of  interfaces  alone  stops  far  short  of  satisfying  the 
needs  of  users  of  critical  systems.  Such  systems  need  to  he  equipped  with  precise  specifications 
of  their  semantics  and  guaranteed  behavior.  When  a  system  is  built  of  smaller  components,  it  is 
important  to  specify  the  properties  of  the  system  in  terms  of  the  properties  of  its  components. 

We  view  formal  specification  and  analysis  as  valuable  tools  that  should  be  at  the  disposal  of 
the  developers  of  distributed  systems. 

Furthermore,  once  a  system  is  specified,  the  implementation  efforts  are  often  informally  con¬ 
nected  to  the  specification  which  implies  that  the  guarantees  offered  by  the.  formal  tools  may  not 
carry  over  to  the  implementation.  Additionally,  the  deployment  of  the  implementation  on  an  ac¬ 
tual  platform  raises  its  own  set  of  challenges  to  meet  the  timing,  fault- tolerance  and  scalability 
requirements. 

It  is  thus  desirable  to  offer  an  integrated  approach  that  covers  the  entire  process,  from  design 
to  implementation  and  deployment  of  the  resulting  distributed  system. 

Our  approach  to  modeling  and  analysis  of  complex  distributed  systems.  This  project 
developed  techniques  and  tools  that  are  designed  to  be  used  in  constructing  provably  correct  dis¬ 
tributed  software.  At  the  specification  level,  It  leverages  the  IOA  formalism  (named  after  lu- 
put/Outpiit  Automata)  and  its  companion  toolset  Tempo.  IOA  use  mathematical  models— in 
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particular,  interacting  state  machines — as  an  integral  part  of  the  software  development  process. 
The  stages  of  this  process  within  the  scope  of  our  framework  are  as  follows.  Abstract  requirements 
for  a  distributed  system  are  specified  using  a  modeling  language.  These  specifications  arc  then  re¬ 
fined  through  multiple  levels  of  abstraction.  Each  refinement  step  is  formally  validated.  Validation 
techniques  include  a  combination  of  simulation,  model  checking,  and  tl  eorem  proving.  The  goal 
of  the  refinement  process  is  to  produce  sufficiently  detailed  models  that  (a)  can  ultimately  be  used 
to  generate  distributed  code  automatically,  and  that  (b)  are  guaranteed  to  be  consistent  with  the 
modeled  system  requirements. 

To  support  automated  formal  methods  for  constructing  or  analyzing  systems,  a  modeling  lan¬ 
guage  must  rest  on  a  solid  mathematical  foundation.  The  I/O  automaton  model  [20]  and  its  timed 
extensions  [13]  provide  such  a  foundation.  I/O  automata  have  been  used  to  describe  and  verify 
many  distributed  algorithms  and  systems  (see  for  example,  [1G] ),  and  Tuned  I/O  Automata  have 
been  used  to  model  timing-dependent  distributed  algorithms  and  real-time  control  systems  [13]. 

The  Tempo  language  uses  the  Timed  I/O  Automata  to  describe  interacting  slate  machines. 
They  are  nondelerininistic,  which  makes  them  suitable  for  describing  systems  in  their  most  general 
forms.  The  stale  of  a  TIOA  can  change  in  two  ways:  discrete  transitions  which  are  labeled  by 
discrete  actions,  change  the  slate  instantaneously,  whereas  trajectories  are  functions  that  describe 
the  evolution  of  the  state  variables  over  intervals  of  time. 

Target  systems.  Many  types  of  systems  are  currently'  developed  using  software  engineering 
methodology  that  is  less  than  adequate  in  its  ability  to  handle  formal  modeling  and  analysis  of 
complex  distributed  software,  and  we  anticipate  that  several  specific  types  of  systems  will  benefit 
from  being  designed  within  our  proposed  framework.  The  types  of  systems  include: 

•  Distributed  data  systems:  data  collection,  management,  dissemination:  consistent  replicated 
sliared-dala  systems. 

•  Communication:  group  communication  systems,  broadcast  and  multicast  systems  with  qualily- 
of-service  guarantees. 

•  Coordination  and  control:  traffic  management,  industrial  process  control,  automated  manu¬ 
facturing  systems,  transportation  (c.g.,  TCAS,  traffic  collision  avoic.ancc  system  used  In  civil 
aviation). 

Many  such  systems  involve  specialized  distributed  platforms,  such  as  networks  of  sensors  and 
mobile  ad  hoc  networks. 

1.2  Summary  of  project  objectives  and  accomplishments 

Willi  the  ultimate  goal  of  providing  a  more  complete  formal  methodology  and  associated  tools  to 
substantially  improve  the  state  of  the  art  in  developing  software  for  complex  distributed  systems, 
the  project  objectives  encompassed  the  following. 

Theory  and  Methodology.  In  developing  service  definitions  and  algor  thins  for  distributed  sys¬ 
tems,  analyzing  the  resulting  specification,  and  generating  code  from  specifications  for  such  systems, 
the  results  only  make  sense  if  they  are  ultimately  based  on  a  sound  underlying  mathematical  model. 
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This  project  uses  interacting  state  machine  models.  Standard  models  like  I/O  (Iuput/Oulput)  au¬ 
tomata  and  timed  1  /O  automata  provide  the  foundation  for  the  Tempo  language  that  we  develuped 
previously.  However,  some  of  the  systems  require  richer  models  capable  of  describing  complexities 
such  as  probabilistic  and  cunlimious  behaviors.  New  models  for  particular  kinds  of  timing  and  fail¬ 
ure  behavior  and  corresponding  analysis  methods  are  also  needed.  Existing  models  that  handle  such 
features  need  to  be  improved  and  better  integrated.  Other  extensions  are  needed  to  express  con¬ 
straints  for  deploying  systems  defined  in  terms  of  the  Tempo  language  in  physical  target  networks, 
and  for  optimizing  deployment  over  target  networks  based  on  various  performance  considerations. 
We  collectively  refer  to  these  I/O  automata  models  and  the  languages  for  system  specifications 
in  these  models  as  Tempo*.  In  this  project  we  have  advanced  the  theory  for  such  models,  in 
particular  the  probabilistic  extensions,  we  have  extended  the  furnial  framework  and  methodology 
for  analyzing  system  specifications,  and  deriving  distributed  code  from  such  specifications,  and  we 
have  prototyped  languages  for  specifying  complex  distributed  systems  in  such  models. 

Our  accomplishments  in  this  area  are  presented  in  this  report  as  follows.  The  probabilistic 
and  hybrid  extensions  to  the  Inpul/Output  Automata  framework  are  presented  in  Section  3.  The 
deployment-oriented  augmentations  of  the  Tempo  framework  are  presented  in  Section  5. 

Modeling .  Analysis,  Code  Generation ,  and  System  Deployment  Tools.  We  performed 
research  and  feasibility  studies  needed  to  develop  computer-aided  design  tools  for  analysis  of  com¬ 
plex  distributed  systems  expressed  in  the  Tempo*  formalism  and  to  prototype  such  tools  un  the 
basis  of  the  Tempu  framework  developed  previously.  New  modeling  and  analysis  tools  and  tool 
extensions  that  are  the  result  of  our  work  include  the  following.  The  language  processor  is  a  front 
end  tool  that  will  accept  Tempo *  specifications,  perform  static  and  type  analysis,  and  produce 
intermediate  output  for  use  by  other  tools.  The  simulator  is  a  tool  designed  tu  simulate  execu¬ 
tions  of  Tempo*  specifications  and  to  provide  linked  simulations  of  pairs  of  specifications,  where 
one  specification  gives  an  abstract  definition  and  the  other  is  a  more  concrete  specification  that  is 
supposed  to  implement  the  abstract  definition. 

Building  on  our  prior  work  on  code  generation  for  distributed  systems,  we  have  explored  for¬ 
mal  approaches  to  code  generation  from  Tempo*  specifications,  and  prove  theorems  abuut  the 
correctness  of  the  resulting  code.  We  prototyped  tools  for  mapping  Tempo*  system  specifications 
consisting  of  multiple  automata  to  target  networks  subject  to  distributed  deployment  constraints 
and  efficiency  and  resource  considerations,  e.g.,  communication  bandwidth,  sturage  requirements, 
and  redundancy  for  fault-tolerance. 

Our  accomplishments  in  this  area  are  presented  in  this  report  as  follows.  An  overview  of  and 
our  latest  refinements  to  the  existing  Tempo  integrated  development  environment  are  presented 
in  Sections  2  and  4.  The  tools  and  translators  for  dealing  with  deployment  problem  of  systems 
specified  in  Tempo  are  presented  in  Sections  5  aud  6.  In  Section  8  wc  summarize  our  work  on  formal 
treatment  of  channel  implementations  as  a  part  of  our  work  towards  code  generation  extensions. 

Applications:  Evaluations  and  Feasibility.  In  order  to  evaluate  the  effectiveness,  scalability, 
and  extensibility  of  our  methodology  and  prototypes,  we  applied  them  to  model  and  analyze  repre¬ 
sentative  systems.  Compared  to  previous  attempts  to  optimize  the  deployment  of  interesting  sys¬ 
tems  we  have  obtained  substantial  improvements  using  our  integrated  approach  with  constrained- 
programming  based  solutions. 

We  present  out  accomplishments  in  Section  C  and  7. 
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1.3  Project  team  and  academic  partner  institution 

The  project  team  over  the  duration  of  the  effort  included  the  following  people: 

Laurent  Michel,  Ph.D.,  System  Architect  and  PI 
Nancy  Lynch,  Ph.D.,  Chief  Technical  Officer  and  Co-PI 
Alex  Shvartsman,  Ph.D.,  Project  Manager  and  Co-PI 
Carloton  Colfrin,  Senior  Software.  Engineer 
Elaine  Souderegger,  Graduate  Researcher,  Development 
Dilsun  Kaynar,  Tempo  Consultant 

Our  academic  partner  on  this  project  was  the  University  of  Connecticut 

1.4  Publications 

In  this  section  we  list  publications  directly  related  to  the  project  that  were  authored  or  co-authored 

by  the  project  personnel.  All  publications  are  available  on  request. 
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Event  Dynamic  Systems  (DEDS),  Springer,  volume  18,  number  1,  March  2008 

[P2]  Ran  Canetti,  Ling  Cheung,  Dilsun  Kaynar,  Nancy  Lynch,  and  Olivier  Pereira.  Modeling 
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1 9 t.h  International  Conference ,  pages  114  130,  2008. 

(P3j  Cliryssis  Georgiou,  Peter  M.  Musial,  Alexander  A.  Shvartsman,  Elaine  L.  Sondereg- 
ger:  An  Abstract  Channel  Specification  and  an  Algorithm  Implementing  It  Using  Java 
Sockets.  Proceedings  of  The  Seventh  IEEE  International  Symjjosium  on  Networking 
Computing  and  Applications,  NCA  2008 ,  pages  211  210,  2008. 

(P4J  Daniel  Libcrzon,  Sayan  Mitra,  and  Nancy  Lynch.  Verifying  Average  Dwell  Time  of 
Hybrid  Systems.  To  appear  in  ACM  Transactions  in  Embedded  Computing  Systems. 

[P5]  N.  Lynch,  L  Michel,  and  A.  Shvartsman,  “Tempo:  A  Toolkit  for  the  Timed  Input/Out¬ 
put  Automata  Formalism”,  First  International  Conference  on  Simulation  Tools  and 
Techniques  for  Communications ,  Networks  and  Systems  (SlMUToo*s  200S).  Industrial 
Track:  Simulation  Works.  CDROM,  paper  3105,  8  pages,  Marseilles,  France,  March 
4-7,  2008. 

(PG]  L.  Michel.  A.  Shvartsman,  E.  Souderegger  and  P.  Van  Hentenryck,  “Optimal  Deploy¬ 
ment  of  Eventually-Serializable  Data  Services”,  Proceedings  of  the  Fifth  International 
Conference  on  Integration  of  A I  and  OR  Techniques  in  Constraint  Programming  for 
Combinatorial  Optimization  Problems,  CPAIOR  2008 ,  Paris,  France,  May  20-23,  2008. 

[P7]  L.  Michel,  A.  Shvartsman,  E.  Souderegger  and  P.  Van  Hentenryck  “Optimal  Deployment 
of  Eventually-Serializable  Data  Services”,  Submitted  to  Annals  of  Operations  Research , 
October,  2008. 

(The  complete  bibliography  cited  in  this  report  is  included  after  the  main  text.)  * 
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1.6  Document  structure 

Section  2  presents  the  overall  framework  at  a  high  level,  including  tools  descriptions,  and  spedliea- 
tioii  examples.  In  the  rest  of  the  report  we  describe  in  detail  the  Phase  1  work  and  onr  accomplish¬ 
ments.  In  Section  3  we  describe  the  advances  in  extending  the  formal  Input/Output  Automata  and 
Timed  Input/Output  Automata  frameworks  to  include  reasoning  about  probabilistic  and  hybrid 
systems.  In  Section  4  we  first  briefly  review  the  architecture  of  the  integrated  Tempo  framework 
and  focus  in  section  5.1  on  the  new  annotations  related  to  deployment  issues.  In  Section  6  we 
present  the  translation  module  responsible  for  deriving  combinatorial  optimization  models  from 
Tempo  specifications.  In  Section  7  we  discuss  the  optimizer  back-end  and  illustrate  its  capabilities 
on  the  deployment  of  a  distributed  system  (Eventually  Serializable  Data  Services).  In  Section  8  we 
summarize  our  work  on  formal  treatment  of  channel  implementations  as  a  part  of  our  work  towards* 
code  generation  extensions. 

We  conclude  in  Section  9.  Bibliography  completes  this  report. 

2  Tempo  Toolkit  for  Timed  Input/ Output  Automata  Formalism 

Tempo  is  a  formal  language  for  modeling  distributed,  concurrent  and  timed  systems  as  collections 
of  interacting  state  machines,  called  timed  input/output  automata.  Tempo  provides  natural  math¬ 
ematical  notations  for  describing  systems,  their  intended  properties,  and  intended  relationships 
between  their  descriptions  at  varying  levels  of  abstraction.  The  Tempo  Toolkit  is  an  implementa¬ 
tion  of  the  Tempo  language  and  a  suite  of  tools  that  supports  a  range  of  validation  methods  for 
descriptions  of  systems  and  their  properties,  including  static  analysis,  simulation,  and  machine- 
checked  proofs.  This  section  gives  an  overview  of  the  Tempo  language  and  illustrates  its  utility  on 
selected  examples  of  importance  to  distributed  computing.  The  focus  of  the  presentation  is  on  the 
Tempo  tools.  We  quickly  review  the  purpose  of  Timed  I/O  Automata  and  TEMPO  language  2.1, 
the  Tempo  toolset.  (Section  2.2),  and  briefly  review  an  example  in  Section  2.3. 

2.1  What  is  the  Tempo  language? 

Tempo  is  a  formal  language  for  modeling  distributed  systems  as  collections  of  interacting  state 
machines  called  Timed  Input/Output  Automata  [13].  Timed  Input/OulpuL  Automata  are  often 
referred  to  as  Timed  I/O  Automata,  or  just  TIOAs.  The  distributed  systems  in  question  may  have 
timing  constraints,  for  example,  bounds  on  the  time  when  certain  events  may  occur,  or  bounds  on 
the  rates  of  change  of  component  clocks.  They  may  use  time  in  significant  ways,  for  example,  for 
timeouts,  or  for  scheduling  events  to  occur  periodically.  Timed  I/O  Automata  formalism  provides 
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good  support  for  describing  these  constraints  and  capabilities.  Timed  anc  untimed  I/O  Automata 
formalisms  have  been  effectively  used  tor  specifying  numerous  distributed  a  id  concurrent  algorithms 
[16].  The  Tempo  language  provides  simple  formal  notation  for  describii  g  Timed  I/O  Automata 
precisely,  based  on  the  pseudocode  notation  that  lias  been  used  in  many  research  papers.  It  also 
allows  specification  of  properties  such  as  invariant  assertions  and  rclatioi  ships  between  automata 
at  different  levels  of  abstraction.  The  Tempo  language  is  supported  by  an  associated  integrated 
development  environment  toolkit,  also  called  Tempo,  that  provides  an  eirtensible  framework  sup¬ 
porting  a  range  of  integrated  analysis  and  validation  tools,  including  static  analysis,  simulation, 
model-checking,  and  theorem-proving. 

Many  distributed  systems  involve  a  combination  of  computer  components  and  real-world,  phys¬ 
ical  entities  such  as  vehicles,  robots,  or  medical  devices.  Systems  involving  interaction  between 
computer  and  real-world  components  usually  have  strong  safety,  reliability,  and  predictability  re¬ 
quirements,  stemming  from  the  requirements  of  real-world  applications.  This  makes  it  especially 
important  to  have  good  methods  for  modeling  the  systems  precisely  and  analyzing  their  behavior 
rigorously.  Tempo  provides  a  simple,  elegant,  and  powerful  mathematical  foundation  for  analyz¬ 
ing  a  wide  variety  of  systems,  and  it  can  be  used  to  model  both  computer  and  real-world  system 
components,  as  well  as  their  interactions. 

Tempo  can  be  used  to  model  practically  any  type  of  distributed  system,  including  (wired  and 
wireless)  communication  systems,  real-time  operating  systems,  embedded  systems,  automated  pro¬ 
cess  control  systems,  and  even  biological  systems.  The  behavior  of  these  systems  generally  includes 
both  discrete  state  changes  and  continuous  state  evolution;  Tempo  is  cesigned  to  express  both 
kinds  of  changes. 

The  Tempo  Toolkit  was  developed  by  VeroModo  Inc.,  with  support  provided  by  an  AFOSR 
technology  transfer  grant.  The  beta  releases  of  the  Tempo  Toolkit  for  Linux,  Windows,  and  Mac 
OS  X  platforms  are  available  for  download  at  www.veromodo.com. 

Earlier  work  on  a  toolkit  supporting  S|xx:ificjit ion  in  (untuned)  Inpir. /Output  Automata  was 
performed  at  the  MIT  Theory  of  Distributed  Systems  group  [9].  The  prototype  toolkit  supported 
a  simulator  [6],  paired  automata  simulation  [28],  and  simulations  of  composed  automata  [29]. 

2.2  Tempo  language  overview 

We  now  discuss  the  Timed  I/O  Automata  formalism  that  is  the  basis  of  the  Tempo  language,  and 
summarize  the  capabilities  of  the  toolkit. 

2.2.1  Timed  I/O  Automata 

The  Timed  I/O  Automata  [13]  mathematical  framework  is  an  extension  of  the  classical  I/O  Au¬ 
tomata  framework  [20,  16],  which  for  many  years  has  been  successfully  used  in  the  theoretical 
distributed  computing  research  community  to  specify  and  reason  about  distributed  and  concurrent 
algorithms.  I/O  Automata  are  very  simple  interacting  asynchronous  state  machines,  without  any 
support  for  describing  timing  features.  Although  they  are  simple,  I/O  Au  mnala  provide  a  rich  set 
of  capabilities  for  modeling  and  analyzing  distributed  algorithms.  I/O  Automata  support  descrip¬ 
tion  of  many  properties  that  distributed  algorithms  are  required  to  satisfy,  and  mathematical  proofs 
that  the  algorithms  in  fact  satisfy  their  required  properties.  These  proofs  are  based  on  methods 
such  as  im'arianl  assertions  and  compositional  reasoning.  I/O  Automata  also  support  representa¬ 
tion  of  algorithms  at  different  levels  of  abstraction,  and  proofs  of  consistercy  relationships  between 
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algorithm  representations  at  different  levels.  Because  of  these  capabilities,  I/O  Automata  have 
been  used  fairly  extensively  for  modeling  and  analyzing  asynchronous  distributed  algorithms,  and 
even  for  proving  impossibility  results  about  computability  in  asynchronous  distributed  settings. 

However,  ordinary  I/O  Automata  cannot  be  used  to  describe  distributed  algorithms  Lhat  use 
time  explicitly,  for  example,  those  that  use  timeouts  or  schedule  events  periodically.  And  they  do  not 
provide  explicit  support  for  describing  timing  constraints  such  as  bounds  on  message  delay  or  clock 
rates.  Moreover,  without  support  for  timing,  I/O  Automata  could  not  be  used  for  other  applications 
such  as  practical  communication  protocols.  These  limitations  led  to  the  development  of  Timed 
I/O  Automata,  which  include  new  features — most  notably,  trajectories — specifically  designed  for 
describing  timing  aspects  of  systems. 

Like  ordinary  I/O  Automata.  Timed  I/O  Automata  are  simple  interacting  state  machines  and 
have  a  well-developed,  elegant  theory,  presented  in  (13).  Like  I/O  Automata,  Timed  I/O  Automata 
provide  a  rich  set  of  capabilities  for  system  modeling  and  analysis.  Methods  used  for  analyzing 
timed  I/O  automata  are  essentially  the  same  as  those  used  for  ordinary  I/O  automata:  invariant 
assertions,  compositional  reasoning,  and  correspondences  between  levels  of  abstraction. 

2.2.2  The  Tempo  language  and  tools 

I/O  Automata  and  Timed  I/O  Automata  are  fine  mathematical  modeling  frameworks  for  dis¬ 
tributed  systems  and  have  been  used  by  hand  to  describe  and  analyze  distributed  algorithms, 
communication  protocols,  and  embedded  systems.  Yet,  computer  support  could  make  these  tasks 
quite  a  bit  easier.  The  Tempo  Language  and  Toolkit  is  an  attempt  at  providing  a  broad  set  of  tools 
to  support  these  activities. 

The  Tempo  toolkit  contains  tools  to  support  analysis  of  systems.  These  include  a  compiler  that 
checks  syntax  and  perforin  static  semantic  analysis  a  simulator  to  produce  and  explore  execution 
traces  for  an  automaton;  a  translation  module  to  the  UPPAAt.  model-checker  |14|;  and  a  translation 
module  to  the  PVS  interactive  tlieorein-prover  (27).  The  overall  architecture  of  the  Tempo  toolkit 
has  been  designed  to  facilitate  incorporation  of  other  validation  tools  in  the  future. 

The  Tempo  language  lias  a  rather  minimal  syntax,  which  closely  matches  the  simple  semantics 
of  the  Timed  I/O  Automata  mathematical  framework.  In  fact,  the  mapping  between  a  Tempo 
automaton  description  and  the  Timed  I/O  Automata  that  it  denotes  is  pretty  transparent.  For 
example,  an  automaton’s  discrete  transitions  and  continuous  evolutions  are  described  directly  in 
Tempo,  by  "transitions”  and  “trajectories",  respectively.  The  minimality  of  the  Tempo  language 
does  not  limit  its  expressive  power:  Tempo  is  capable  of  describing  very  general  systems  of  Timed 
I/O  Automata.  Of  course,  many  analysis  tools— especially  automated  ones  like  model-checkers — 
are  not  capable  of  handling  fully  general  Tempo  programs.  In  contrast  with  the  conventional 
approach  taken  by  developers  of  automated  tools.  Tempo  does  not  outright  limit  the  expressive 
power  of  the  language  and  opts  instead  for  the  definition  of  sublanguages  that  are  suitable  for  use 
with  particular  tools. 

2.3  An  Example:  mutual  exclusion  algorithm 

To  illustrate  the  capabilities  of  Tempo  and  its  simulator,  we  will  be  using  the  Fischer  Timed  Mutual 
Exclusion  Algorithm.  It  has  become  famous  as  a  standard  test  example  for  formal  methods  for 
modeling  and  analyzing  timed  systems.  An  informal  description  of  the  example  appears  in  {10], 
Chapter  24. 
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2.3.1  The  Tempo  specification 

This  example  illustrates  most  of  the  basic  constructs  needed  for  writing  a  Tempo  program  for  a 
single  Timed  I/O  Automaton  modeling  a  shared-memory  system.  The  e>  ample  also  demonstrates 
how  to  express  invariants  using  Tempo,  including  invariants  that  involve  time. 

The  Tempo  model  shown  in  Cude  1  and  2  describes  the  entire  syste:  i  as  a  single  Timed  I/O 
Automaton.  The  vocabulary  section  declares  the  data  types  used  in  the  algorithm,  namely,  the 
abstract  data  type  process  and  the  program  counter  abstract  data  type  Pc  Value  (an  enumerated 
type)  to  represent  the  exact  location  of  each  process  in  its  program.  Eac  i  process  could  be  in  its 
remainder  region  (program  counter  =  pc_rem),  where  it  is  not  engaged  in  t*ying  to  enter  the  critical 
region.  Or,  it  could  be  abuut  tu  test,  set,  or  check  the  turn  variable.  C  it  could  be  in  various 
stages  of  entering  or  leaving  the  critical  region  -the  model  uses  separate  p  ogram  counter  values  to 
represent  situations  where  the  process  has  successfully  completed  the  trying  protocol,  where  it  is 
actually  in  the  critical  region,  where  it  is  about  to  reset  the  turn  variable  ipon  leaving,  and  where 
it  has  successfully  completed  the  exit  protocol. 

The  actual  automaton  description  begins  with  the  name  of  the  automaton,  with  formal  param¬ 
eters  Lcheck  and  u.set.  These  are  real  numbers  representing,  respectively,  a  lower  bound  on  the  time 
between  setting  and  checking,  and  an  upper  bound  on  the  time  between  checking  and  setting.  The 
where  clause  specifies  restrictions  imposed  on  the  parameters  saying  (most  importantly)  that  a .set 
must  be  strictly  less  than  Lcheck.  The  automaton  imports  the  vocabulary  to  make  its  definition 
available  to  the  remainder  of  the  specification. 

The  automaton's  signature,  describe  its  actions.  Actions  are  classified  as  input,  output,  or 
internal.  Here,  no  input  actions  are  used,  i.e.,  the  system  is  “closed”.  S  nce  the  entire  system  is 
being  modelled  by  a  single  automaton,  each  type  of  action  is  parameter  -zed  by  the  name  of  the 
process  that  perforins  it.  In  this  model  the  internal  actions  are  associated  with  shared-variable 
accesses — the  steps  that  test,  set,  check,  and  reset  the  turn  variable.  The  output  actions  are  those 
that  mark  processes5  progress  through  the  various  high-level  regions  of  llitr  code:  The  fry(i)  action 
describes  process  i  moving  from  its  remainder  region  to  its  trying  region,  in  which  it  executes  a 
protocol  to  try  to  reach  the  critical  region  The  crit{i)  action  describes  passage  from  the  trying 
region  to  the  critical  region,  and  the  cxit(i)  action  describes  passage  from  die  critical  region  to  the 
exit  legion ,  where  process  i  performs  its  exit  protocol.  Fiually,  the  rem(i)  action  describes  passage 
from  the  exit  region  back  to  the  remainder  region. 

The  automaton’s  state  is  specified  in  the  states  section.  The  shared  variable  turn  has  type 
Null  [process],  which  indicates  that  its  value  can  either  be  a  process  or  the  special  value  nil  to 
indicate  the  absence  of  value,  turn  is  initially  set  to  nil.  The  variable  pc  represents  the  program 
counters  for  all  of  the  processes  in  an  array  of  PcVatuc  indexed  by  processes.  Initially,  all  of  the 
program  counter  values  are  set  to  pc. tern,  which  means  that  all  of  the  processes  start  out  in  the 
remainder  region. 

The  remaining  three  variables  are  introduced  solely  to  express  the  needed  timing  constraints. 
First,  the  variable  now  is  used  to  represent  the  real  time.  It  is  initialized  -l  0. 

Second,  the  variable  lasl.sct  is  an  array  containing  absolute  real  time  i  Dper  bounds  ( deadlines ) 
for  the  processes  to  perform  set  actions.  A  deadline  will  be  in  force  for  a  process  i  only  when  its 
program  counter  is  equal  to  pc.set ,  that  is,  when  it  is  in  fact  ready  to  set  the  turn  variable.  In  this 
case,  the  value  of  /osLscf[i]  will  be  a  nonuegative  real  number;  otherwise,  that  is,  if  the  program 
counter  is  anything  other  than  pc.sct ,  the  value  will  be  oc,  representing  .lie  absence  of  any  such 
deadline.  The  elemeuls  of  the  lasLset  array  are  defined  to  be  of  typed v~mcntcd Ileal:  a  type  that 
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vocabulary  ftschcrUypes 
types  process, 

PcValue  :  Enumeration  [pc_rem,  pc_Ust,  pcscl ,  pc-check, 
pcJcavctry ,  pc-crit,  pc_reset,  pc_lcaveexit\ 

end 

automaton  fischcr[Lchcck,  u.sel  Real) 
where  u.set  <  Lcheck  At l.set  >0  A Lcheck  >0 
imports  fischcrJypcs 

signature 

output  <ry(i:  process) 
output  cnt(i:  procesa) 
output  exit(i:  process ) 
output  nem(i:  proceas) 
internal  ieat(i:  proeesa) 
internal  sct(i:  process) 
internal  check(i.  process) 

Internal  resci(i:  process ) 

states 

turn:  Null[proccss]  :  =  nit ; 

pc:  Ar  ray  [process,  Pc  Value )  :  =  coftsfa»f(pc_nem); 
now.  Real :  =  0; 

lasLset-  Arrayfprocess,  AugmenlcdRcat\  :  =  constanifoo), 

Jirst.check  Ar  ray  [procesa,  DiscreteReal)  .  =  constant(  0), 

transitions 
output  try(t) 
pro  pc[i]  =pc_nem; 
off  pc[i] :  =  pc  Jest; 
internal  <esf(i) 
pre  pc[t)  =pedea<; 
eff  if  turn  =nil  then 
pc[t] :  =  pc.sel j 
laa/_aci[i]  :  =  (note  +  ti.sel); 
fi; 

internal  set(i) 
pre  pc[i]  =  pc-set; 
eff  !urn  :  =  embcd(i)‘t 
pc[t)  :  =  pc.check ; 
iasi-aci[t] ;  =  oo; 

Jirsl.check[{\ :  =  now  +  Lcheck, 

Code  1:  Tempo  spec,  of  the  Fischer  algorithm  (I) 
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Internal  check(i) 

pro  pe[ij  =pc^check  AfirsLcheck\i\  <now, 
off  if  turn  =cmbed(i )  then 
pc[i]  :  =  pc_leavehy, 
else 

peft]  :  =  pc.test; 

fi; 

ftraLcheck\i)  :  =  0; 
output  crif(i) 
pro  pc[i]  =pc.leavetry, 
off  pc[t]  :  as  pc.crit; 
output  exit(i) 
pro  pc[i)  =pc.cnt; 
efT  pc[i'j  :  =  pc-rc$et\ 
internal  rescf(i) 
pro  pc[i ]  —pc^reset; 
off  pc[i]  :  =  pc-lcavecxit-, 
turn  :  =  nil\ 
output  rem(i) 
pro  pc[i]  =pc.teaveexit, 
off  pc[i]  :  =  pc~rem\ 
trajectories 
trajdof  traj 
stop  when 

3i:  process  ( now  =/ajL3et[t]); 
evolve 
d(now)  *sl; 


Code  2:  Tempo  spec,  of  the  Fischer  algorithm  (I!) 
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includes  all  (positive  and  negative)  real  numbers  plus  two  values  corresponding  to  positive  and 
negative  infinity.  Initially,  since  none  of  the  program  counters  is  pc-sct,  the  values  in  the  array  are 
all  oo. 

Third  and  finally,  the  variable  first-check  is  an  array  containing  absolute  real  time  lower  bounds 
( earliest  limes)  for  the  processes  to  perforin  check  actions,  when  their  program  counters  are  equal 
to  pc.check.  The  elements  of  firsLcheck  are  of  type  Discrclcllcal ,  which  means  that  they  always  have 
Ileal  values,  aud  moreover,  they  do  not  change  between  discrete  actions. 

The  detailed  description  of  the  transitions  of  the  automaton  follows  in  the  transitions  section. 
Transitions  are  (state,  action,  state)  triples.  The  transitions  are  described  in  guarded  command 
style,  using  small  pieces  of  code  called  tmnsition  definitions.  Each  transition  definition  denotes  a 
collection  of  transitions,  all  of  which  share  a  common  action  name. 

Each  transition  has  a  name,  list  of  parameters,  a  precondition  that  indicates  when  the  action  is 
enabled  and  finally,  an  effect  clause  that  describes  the  changes  to  the  state  when  that  accompany  the 
action.  Input  actions  are  always  enabled,  reflecting  the  assumption  that  Timed  I/O  Automata  are 
inpul -enabled.  Notionally,  input  actions  have  no  preconditions,  as  a  shorthand  for  the  precondition 
being  true. 

The  h'yi i)  transition  represents  an  entrance  by  process  t  into  its  trying  region.  The  transition 
is  allowed  to  occur  whenever  pr[i]  = pc.rcm ,  that  is,  whenever  process  i  is*  in  its  remainder  region. 
The  elfect  is  simply  to  advance  the  program  counter  to  pc.Je.st  to  indicate  that  process  t  is  ready  to 
test  the  turn  variable 

The  tcst( i)  transition  represents  process  i  testing  the  turn  variable.  It  is*  allowed  to  occur  when¬ 
ever  pc[i)  =pc-tcst.  The  transition  can  either  find  the  turn  variable  equal  to  m/  at  which  point  it 
moves  to  take  the  turn  (by  setting  the  program  counter  to  pcsct)  and  saves  in  /asf_5ef[tj  the  deadline 
for  the  set  action  to  occur  at  the  latest  in  t uset  time  steps  in  the  future  (away  from  now).  The 
transition  can  also  find  that  turn  is  not  ml  and  simply  takes  no  action  to  remain  in  the  state,  ready 
to  test  again. 

The  sct(i)  transition  represents  process  t  setting  the  turn  variable  to  its  own  index.  This  is 
allowed  to  occur  whenever  />c(i]  —pc.set.  The  effects  arc  given  as  straight-line  code  in  which  process 
i  simply  sets  turn  to  its  own  index  (the  embed  call  is  necessary  to  store  the  value  into  an  object 
of  type  Nnll(pivcess)).  The  code  then  sets  the  program  counter  to  pc-chcck  to  enable  the  c/icct{») 
transition  that  will  verify  the  turn  variable.  Now  that  the  sct(t')  action  lias  occurred,  the  la$t.sei\i] 
deadline  is  reset  to  its  default  value,  oo.  The  code  also  records  the  earliest  time  when  process  t 
could  recheck  the  turn  variable  based  on  the  current  clock  now  and  the  lower  bound  l.check. 

The  check(i)  transition  is  enabled  when  process  i’s  program  counter  is  set  to  pc-chcck  and  its 
earliest  checking  time  has  passed  (first. chcck[i]  <now).  When  the  transition  executes,  two  interesting 
cases  may  arise:  If  process  i  finds  that  turn  is  still  equal  to  t,  it  leaves  the  trying  region  and  enters 
the  critical  region.  Oil  the  other  hand,  if  it  finds  the  turn  variable  equal  to  anything  else,  it  gives 
up  the  current  attempt  and  goes  back  to  the  testing  step.  In  either  case.  first.chcck\{\  is  reset  to  its 
default,  0. 

The  subsequent  transitions  are  quite  straightforward.  A  crii(i)  transition  represents  process  t 
moving  into  the  critical  region  and  an  exif(i)  transition  represents  process  i  leaving  the  critical 
region.  A  resef(i)  transition  represents  process  i  resetting  the  fura  variable  to  its  default  value  nil, 
and  a  rc»n(i)  transition  represents  process  i  returning  to  its  remainder  region. 

The  final  part  of  the  automaton  description  is  the  set  of  trajectories  that  is,  the  functions  from 
time  to  states  that  describe  how  the  state  is  permitted  to  evolve  between  discrete  steps.  This  model 
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specifics  one  trajectory  definition,  named  tvaj.  This  definition  describes  die  evolution  of  the  state 
in  a  way  that  allowed  the  current  time  how  to  increase  at  rate  1.  All  of  the  other  state  variables 
nrc  of  types  that  are  defined  to  be  discrete;  these,  by  default,  are  not  allowed  to  change  during 
trajectories.  The  stop  when  condition  says  that  a  trajectory  must  stop  f  the  slate  ever  reaches  a 
point  where  the  current  time  now  is  equal  to  a  specified  deadline  lastsct[ ),  for  any  i.  That  is  time 
is  not  “allowed  to  pass”  beyond  any  deadline  currently  in  force. 

This  stop  when  condition  is  an  example  of  a  phenomenon  whereby  an  automaton  can  prevent 
the  passage  ol' time.  This  may  look  st range  (at  first)  to  some  programmers,  since  programs  of 
course  cannot  prevent  time  from  passing.  However,  appearances  can  be  deceiving  and  the  Fischer 
automaton  is  not  exactly  a  program;  it  is  a  descriptive  model  that  expresses  both  the  usual  sort  of 
behavior  expressed  bv  a  program,  plus  additional  timing  assumptions  that  might  be  expressed  in 
other  ways. 

2.3.2  Properties  of  the  algorithm 

Tempo  can  be  used  to  describe  not  just  algorithms,  but  also  properties  that  we  would  like  the 
algorithms  to  satisfy.  For  example,  the  Fischer  algorithm  is  supposed  to  satisfy  the  mutual  exclusion 
property,  saying  that  no  two  processes  can  simultaneously  reside  in  their  critical  regions.  This  is  a 
claim  that  the  mutual  exclusion  is  an  inven'iunt  of  the  Fischer  algorithm,  that  is,  that  it  is  true  in 
all  reachable  slates  of  the  fischcr  automaton.  This  claim  can  be  expressed  in  Tempo  with  a  block 

invariant  of  fiseker. 

Vi:  process  Vj:  process 

(i  ^ j  =>(pc[i]  j£pc_crit  Vpc(jj  ^pc_cri/)); 

This  invariant  definition  claims  that,  in  any  reachable  slate  of  the  automaton,  any  two  processes 
cannot  simultaneously  be  in  the  critical  section.  This  formal  statement  must,  of  course  be  verified 
with  a  tool  in  order  to  formally  prove  that  the  algorithm  is  correct.  For  nslance,  one  could  use  an 
interactive  theorem  prover  such  ’as  PVS,  a  model-checker  like  UPPAAL,  or  run  simulations  of  the 
protocol  and  require  the  simulator  to  check  the  assertions  after  every  single  step  of  the  simulations. 

In  the  next  sections  we  describe  the  Tempo  language  and  integratec  development  framework, 
and  their  design  in  more  detail,  and  we  describe  the  work  carried  out  in  Phase  II. 

3  Extensions  of  I/O  Automata  and  Timed  I/O  Automata  Frame¬ 
works 

We  have  continued  our  work  on  mathematical  foundations  for  model. ng  and  analyzing  timed, 
hybrid,  and  probabilistic  systems.  We  have  been  pursuing  an  effort  to  extend  Timed  1/0  Automata 
to  allow  probabilistic  behavior  even  before  the  start  of  Phase  I  work,  resulting  in  several  papers, 
e.g.,  (23,  25.  19].  In  an  extended  series  of  case  studies,  we  have  also  been  using  probabilistic 
(PIOA)  and  timed  I/O  automata  (TIOA)  to  model  and  verify  security  piotocols.  This  has  entailed 
extending  the  formal  foundations  in  several  directions,  to  restrict  possib  lilies  for  nondelerminisin, 
to  define  appropriate  implementation  relationships  for  the.  security  setliig,  and  to  integrate  timing 
into  security  models. 
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3.1  Probabilistic  extensions 

Our  recent  work  investigated  the  time-bounded  task-PIOA  modeling  framework,  an  extension  of 
the  probabilistic  input/output  automata  (PlOA)  framework  that  can  be  used  for  modeling  and 
verifying  security  protocols.  Time-bounded  task-PlOAs  can  describe  probabilistic  and  noudeler- 
ministic  behavior,  as  well  as  tiinebounded  computation.  Together,  these  features  support  modeling 
of  important  aspects  of  security  protocols,  including  secrecy  requirements  and  limitations  on  the 
computational  power  of  adversarial  parties.  They  also  support  security  protocol  verification  using 
methods  that  are  compatible  with  less  formal  approaches  used  in  the  computational  cryptography 
research  community.  We  illustrate  the  use  of  our  framework  by  outlining  a  proof  of  functional 
correctness  and  security  properties  for  a  well-known  oblivions  transfer  protocol.  These  results 
appeared  in  print  in  2008  [4]. 

We  also  introduced  the  notion  of  approximate  implementations  for  Probabilistic  I/O  Automata 
(PIOA)  and  developed  methods  for  proving  such  relationships  (24).  We  eaiploy  a  task  structure 
on  the  locally  controlled  actions  and  a  task  scheduler  to  resolve  uondeterininisni.  The  interaction 
between  a  scheduler  and  an  automaton  gives  rise  to  a  trace  distribution  a  probability  distribution 
over  the  set  of  traces.  We  define  a  PIOA  to  be  a  (discounted)  approximate  implementation  of 
another  PIOA  if  the  set  of  trace  distributions  produced  by  the  first  is  dose  to  that  of  the  latter, 
where  closeness  is  measured  by  the  (resp.  discounted)  uniform  metric  over  trace  distributions.  We 
propose  simulation  functions  for  proving  approximate  implementations  corresponding  to  each  of 
the  above  types  of  approximate  implementation  relations.  Since  onr  notion  of  similarity  of  traces 
is  based  on  a  metric  on  trace  distributions,  we  do  not  require  the  state  spaces  nor  the  space  of 
external  actions  of  the  automata  to  be  metric  spaces.  We  discuss  applications  of  approximate 
implementations  to  verification  of  probabilistic  safety  and  termination. 

3.2  Extensions  for  reasoning  about  security  protocols 

In  another  recent  development  we  investigated  a  new  paradigm  for  the  analysis  of  long-lived  security 
protocols.  We  allow  entities  to  be  active  for  a  potentially  unbounded  amonal  of  real  time,  provided 
they  perform  only  a  polynomial  amount  of  work  per  unit  real  time.  Moreover,  the  space  used  by 
these  entities  is  allocated  dynamically  and  must  be  polynomially  bounded.  We  proposed  a  key 
notion  of  long-term  implementation  which  is  an  adaptation  of  computational  indisLinguishabilily 
to  the  long-lived  setting.  We  show  that  long-term  implementation  is  preserved  under  polynomial 
parallel  composition  and  exponential  sequential  composition.  To  illustrate  the  use  of  this  new 
paradigm,  we  analyze  the  long-lived  timestamping  protocol  of  Haber  and  Kamat.  This  work  was 
submitted  for  publication  in  2008  [5] 

3.3  Extensions  for  hybrid  systems 

We  completed  our  work  on  a  journal  paper  on  average  dwell  time  for  hybrid  systems  [15].  Average 
dwell  time  ( ADT)  properties  characterize  the  rate  at  which  a  hybrid  system  perforins  mode  switches. 
In  this  paper,  we  present  a  set  of  techniques  for  verifying  ADT .  properties.  The  stability  of  a 
hybrid  system  A  can  be  verified  by  combining  these  techaiques  with  standard  methods  for  checking 
stability  of  the  individual  modes  of  A.  We  introduce  a  new  type  of  simulation  relation  for  hybrid 
automata  switching  simulation  for  establishing  that  a  given  automaton  A  switches  more  rapidly 
than  another  automaton  D.  We  show  that  the  question  of  whether  a  given  hybrid  automaton  has 
ADT  a  can  be  answered  either  by  checking  an  invariant  or  by  solving  an  optimization  problem.  For 
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classes  of  hybrid  automata  for  which  invariants  can  be  checked  automat  cally,  the  invariant-based 
method  yields  an  automatic  method  for  verifying  ADT;  for  automata  that  are  outside  this  class 
the  invariant  has  to  be  checked  using  inductive  techniques.  The  optimization-based  method  is 
automatic  and  is  applicable  to  a  restricted  class  of  initialized  hybrid  automata.  A  solution  of  the 
optimization  problem  either  gives  a  counterexample  execution  that  violates  the  ADT  property,  or 
it  confirms  that  the  automaton  indeed  satisfies  the  property.  The  optimization  and  the  invariant- 
based  methods  can  be  used  in  combination  to  find  the  unknown  ADT  of  c  given  hybrid  automaton. 

We  developed  a  new  abstraction  technique,  event  order  abstraction  (EOA),  for  parametric 
safety  verification  of  real-time  systems  in  which  “correct  orderings  of  events"  needed  for  system 
correctness  are  maintained  by  timing  constraints  on  the  systems'  behavio*  [32].  By  using  EOA,  one 
can  separate,  the  task  of  verifying  a  real-time  system  into  two  parts:  l  Safety  property  verification 
of  the  system  given  that  only  correct  event  orderings  occur;  and  2.  Derivation  of  timing  parameter 
constraints  for  correct  orderings  of  events  in  the  system.  The  user  first  ident  ifies  a  candidate  set  of 
bad  event  orders.  Then,  by  using  ordinary  untuned  model-cheeking,  the  user  examines  whether  a 
discretized  system  model  in  which  all  liming  constraints  are  abstracted  away  satisfies  a  desirable 
safety  property  under  the  assumption  (lint,  the  identified  bad  event  outers  occur  in  no  system 
execution.  The  user  uses  counterexamples  obtained  from  the  model  checker  to  identify  additional 
bad  event  orders,  and  repeals  the  process  until  the  model-checking  succeeds  In  this  step  the  user 
obtains  a  sullicicnt  set  of  bad  event  orders  tlmt  must  be  excluded  by  timing  synthesis  for  system 
correctness.  Next,  the  algorithm  presented  in  the  paper  automatically  derives  a  set  of  timing 
parameter  constraints  under  which  the  system  does  not.  exhibit  the  identified  had  event  orderings. 
From  this  step  combined  with  the  uulimed  model-checking  slop  the  ns-?r  obtains  a  sufficient  set 
of  timing  parameter  constraints  under  which  the  system  executes  correctly  with  respect  to  a  given 
safety  property.  In  our  documented  work  we  illustrated  the  use  of  EOA  with  a  train-gale  example 
inspired  by  the  general  railroad  crossing  problem.  We  also  summarized  three  other  case  studies,  a 
biphase  mark  protocol,  the  IEEE  1394  root  contention  protocol,  and  the  Fischer  mutual  exclusion 
algorithm. 

4  Tempo  Toolkit:  Architecture  and  Language 

The  Phase  II  STTR  [18]  completed  in  2007  produces  a  solid  implementation  of  the  TlOA  language 
in  the  form  of  a  toolkit:  Tempo.  VeroModo  focused  on  a  redesign  of  the  core  implementation 
of  the  front-end  (analyzer  and  compiler)  and  a  design  of  its  interfaces  .o  the  various  back-ends. 
Tempo  lias  the  following  characteristics 

•  It  is  a  Java  1.5  implementation  of  a  refined  TIOA  language. 

•  It  offers  a  modular  design  to  facilitate  the  integration  of  additional  tools  as  independent  back¬ 
ends.  (e.g.,  the  PVS  translator,  the  simulator  or  the  model-clieckei).  It  is  based  on  modern 
modular  architecture  where  each  back-end  tool  is  a  plug-in  that  can  be  loaded  at  runtime  to 
extend  the  compiler. 

•  It  features  a  fine-grained  interface  to  communicate  with  back-end  tools  that  would  make  it 
possible  to  establish  a one-to-one  correspondence  between  each  back-end  tool  and  the  TIOA 
abstractions  offered  by  TEMPO. 
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•  it  offers  a  flexible  integration  with  the  back-ends  that  let  each  back-end  independently  re¬ 
fine  the  semantic  rules  to  either  augment  the  core  language  with  back-end  specific  language 
extensions. 

4.1  The  Architecture  of  Tempo 


Figure  1  Tempo’s  architecture 

Tempo’s  architecture  was  designed  and  developed  by  our  academic  partner  Laurent  Michel 
(University  of  Connecticut),  and  it  is  based  on  a  modular  multi-stage  compiler.  The  overall 
organization  is  shown  in  Figure  1.  The  first  two  compiler  stages  are  fixed  and  independent  of  the 
selected  back-end  tool.  The  third  stage  depends  upon  the  selected  tool  and  is  loaded  automatically 
from  a  JAVA  shared  library  (JAR  file)  based  on  the  user  selection  at  the  command  line  or  in  the 
user  interface. 

The  initial  stage  is  responsible  for  the  lexical  and  syntactic  analysis  of  a  Tempo  specification.  It 
assembles  its  input  from  one  or  more,  text  files  containing  the  specifications  as  well  as  one  or  more 
vocubtilurics.  A  vocabulary  is  a  Tempo  specification  containing  built-in  abstract  data  types  for 
commonly  used  data  structures  such  as  sets,  multi-sets,  maps  or  arrays1.  Lexical  and  grammatical 
errors  are  reported  immediately.  The  parser  is  written  with  a  state-of-the-art  freely  available  parser 
generator:  ANTLR  v2.7.x2.  The  output  of  this  phase  is  an  abstract  syntax  tree  that  is  passed  down 
to  a  second  analysis  stage. 

The  second  stage  focuses  on  the  semantic  analysis  of  the  specification.  This  phase  perforins 
multiple  passes  (traversals)  of  the  AST  to  analyze  it. 

From  a  high-level  standpoint,  the  semantic  analysis  applies  a  collection  of  validation  rule  to 
each  node  of  the  abstract  syntax  tree.  Each  rule  take  the  form 


F(n)  =>  9ir  -  ,(Jk 

*Usei5  call  define  their  own  additional  vocabularies  which  are  indistinguishable  from  Tempo's  own  buill-in  vocab¬ 
ularies 

2 ANTLR  v2  is  available  from  http://www.antlr2.org/ 
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where  P  is  a  boolean  predicate  on  the  subtree  rooted  at  n  that  determines  whether  the  rule  is 
applicable  and  <j\  through  i y*  are  actions  that  transform,  annotate,  or  possibly  tag  the  tree  as 
semantically  unsound.  Each  back-end  can  add  a  set  of  validation  rules  that  capture  additional 
requirements  on  the  AST  to  comply  with  its  limitations.  For  instance,  if  a  back-end  cannot  operate 
on  expressions  containing  quantifiers  (V  or  3),  it  can  add  a  rule 

cluss(n)  =  AST Forull  V  chiss(n)  =  ASTExist 

=>  rejccJ(n,  “The  back-end  xyz  does  not  support  quantifiers  in  expressions") 

that  the  semantic  analysis  will  apply  inductively  (alongside  all  the  other  rules)  to  all  the  nodes  of 
the  abstract  syntax  tree. 

4.2  Tempo  language 

The  TIOA  formalism  and  associated  theory  is  defined  in  the  monograph  produced  and  published 
as  a  part  of  this  project  (13j.  We  refer  the  reader  to  the  monograph  for  the  detailed  information 
about  the  TIOA  formalism,  and  modeling  and  analysis  methodology. 

The  TIOA  language  was  refined  during  the  implementation  of  TEMPO  to  take  into  account 
standard  user  expectations  and  to  produce  an  implementation  as  uniform  as  possible.  The  Tempo 
language  itself  has  a  minimal  syntax,  which  closely  matches  the  simple  semantics  of  the  Timed  I/O 
Automata  mathematical  framework.  In  fact,  the  mapping  between  a  Tempo  automaton  description 
and  the  timed  I/O  automaton  that  it  denotes  is  pretty  transparent.  For  distance,  an  automaton's 
discrete  transitions  and  continuous  evolutions  are  described  directly  in  Tempo  by  “transitions”  and 
“trajectories’,  respectively.  The  minimality  of  the  language  does  not  Unit  its  expressive  power: 
Tempo  can  describe  very  general  systems  of  timed  I/O  automata.  Of  course,  each  analysis  tool 
brings  its  own  computational  limitations,  and  Tempo  accommodates  tl  em  with  the  addition  of 
tool  specific  restrictions  (captured  through  the  predicate  mechanism  described  above)  to  define  a 
suitable  sublanguage. 

5  Deployment  Problems 

This  section  reviews  the  deployment  phase  that  arise  when  constructing  a  distributed  systems.  We 
discuss  our  prior  work  in  the  area,  then  present  the  language  extensions  leeded  to  convey  the  key 
characteristics  of  deployment  instances,  and  illustrate  an  application  of  our  framework  in  specifying 
a  meaningful  sample  deployment  problem. 

Our  earlier  work  on  deployment  of  distributed  systems  was  done  in  the  context  of  an  architec¬ 
tural  specification  framework  called  the  X5  [2  lj.  X5  uses  five  levels  of  abstraction,  called  Interface. 
Implementation,  Integration,  Instantiation,  and  Installation,  to  describe  the  hardware  and  soft¬ 
ware  structures  of  distributed  systems.  Deployment  of  software  compo  lents  to  hardware  nodes 
takes  place  at  the  Installation  level  using  information  gathered  at  higher  levels.  X5  does  not  in¬ 
corporate  specification  of  component  semantics,  and  we  explored  the  use  of  the  I/O  Automata 
language  in  [l]  to  complement  the  structural  specifications  in  X\  Specification  of  systems  in  X5 
can  be  done  using  UML,  but  it  is  not  supported  by  au  integrated  development  environment.  The 
deployment  optimization  was  performed  using  customized  techniques  based  on  binary  integer  pro¬ 
gramming  aud  genetic  algorithms  [3).  Our  current  work  on  deployment  op  .imizatiou  in  Tempo  is  in 
part  motivated  by  X5.  By  contrast,  Tempo  provides  an  integrated  development  environment  that 
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incorporates  both  structural  system  descriptions  and  system  semantics,  and  allows  for  automatic 
generation  of  deployment  mappings  using  advanced  constraint-programming  techniques. 

5.1  Augmenting  Tempo  with  deployment  annotations 

The  Tempo  deployment  annotations,  if  any,  arc  part  of  the  definition  of  a  composite  automaton. 
The  composite  automaton  is  the  only  portion  of  a  TlOA  model  which  has  multiple  components, 
and  it  is  these  component  parts  which  potentially  could  be  deployed  to  different  computing  nodes. 

The  simple  composite  automaton  in  Figure  2  illustrates  the  required  deployment  annotations. 
(Section  5.3  has  a  more  realistic  example  using  an  Eventually  Serializable  Data  Service  (ESDS)  (8, 
7].)  Our  example  composite  automaton  consists  of  two  types  of  components,  A  and  D.  Automaton 
A  has  two  output  transitions,  a  send  transition  that  specifies  both  a  message  to  be  sent  and  the 
identifier  of  its  destination  and  a  gossip  transition  that  specifics  data  to  be  broadcast.  Automaton  D 
has  the  two  matching  input  transitions.  For  simplicity,  the  state  and  transition  details  for  automata 
A  and  D  have  been  omitted.  The  composite  automaton  C  has  two  instances  of  automaton  A,  called 
«1  and  «2,  and  three  instances  of  automaton  D.  called  61,  62,  and  63. 


automaton  A 
signature 

output  send(m  :  String,  id  .  Nat) 
output  gossipfdritfl  ArrayfNat  Nat)) 
states 
transitions 
output  send{m, id) 
output  gossip(dntn) 

automaton  U{id  :  Nat) 
signature 

input  send{ni :  String  const  id) 
input  gossip{da(a  :  ArrayjNat,  Nat)) 
states 
transitions 
input  sen d(m  id) 
input  gossip(data) 


automaton  C 
components 
al  :  A\ 

«2  :  A; 

61  :  if(l) 

62  : 13( 2); 

63  .  13( 3), 

deployment 

nodes 

nl; 

ti2; 

n3; 

h4 


connections 
{nl,  n2); 

{n2,  u3,  »i4); 

communication 
<il. gossip  ->  61,62,63  freq  5; 
«2.send  ->  63  freq  10; 
«2.send  ->  61  freq  2; 


Figure  2:  Simple  composite  automaton  with  deployment  annotations 

The  deployment  annotations  begin  with  the  keyword  deployment  and  contain,  at  a  minimum,  a 
list  of  the  computing  nodes,  the  physical  connections  among  those  nodes,  and  a  description  of  the 
communication  patterns  of  the  composite  automaton’s  components.  The  list  of  computing  nodes 
begins  with  the  keyword  nodes  and  is  followed  by  the  list  of  all  the  computers  in  the  network, 
namely  nl,  n2,  n3,  and  n4.  lu  this  example,  node  nl  is  directly  connected  to  n2,  and  nodes  u2, 
n3,  and  n4  are  directly  connected  to  each  other  by  a  common  connector.  This  is  denoted  by  the 
deployment  section  beginning  with  the  keyword  connections  and  containing,  for  each  set  of  directly 
connected  nodes,  a  list  of  the  individual  nodes,  separated  by  commas  and  enclosed  in  braces. 

The  last  deployment  section  in  this  example  begins  with  the  keyword  communication  and  lists 
the  relative  frequencies  with  which  each  of  the  transitions  of  the  composite  automaton  occur  For 
each  transition,  the  component  which  generates  the  transition  as  an  output  transition  is  listed 
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first,  followed  by  a  period,  the  name  of  the  transition,  the  symbol  and  the  names  of  all  the 
components  which  receive  this  transition  as  input.  The  list  of  input  co.  lponenls  are  followed  by 
the  keyword  freq  and  an  expression  for  Lhe  relative  frequency  of  the  transition.  Each  frequency 
expression  is  interpreted  as  the  number  of  times  the  transition  occurs  during  some  time  period 
of  unspecified  length,  where  it  is  assumed  that  the  same  time  period  is  used  in  determining  the 
frequencies  for  all  the  transitions.  Here,  component  «1  broadcasts  its  gossb  message  to  components 
61,  62.  and  63  with  a  relative  frequency  of  5  while  «2  outputs  its  send  lessage  to  component  63 
with  a  relative  frequency  of  10. 

Figure  3  enhances  our  example  composite  automaton  with  some  of  the  optional  deployment 
constraints.  The  first  constraint,  beginning  with  the  keyword  support,  specifics  which  components 
may  run  on  which  nodes.  In  our  example,  node  nl  supports  components  «1  and  a2,  node  n2 
supports  all  the  components,  and  nodes  7i3  and  u4  support  components  61,  62,  and  63.  If  no 
support  section  is  provided  in  the  deployment  annotations,  every  compc  lent  may  be  deployed  to 
every  node. 


automaton  C 

connections 

separated 

components 

{nl,?i2}; 

{cl  a2}; 

cl  :  A; 

{n2,«3,  n4}; 

{61,62,63] 

c2  :  A\ 

61  :  B; 

support 

together 

62:  B, 

nl  <-  cl. o2; 

{cl,  61}; 

63: 

n2  <-  all; 

r?3  <-  61  62,63; 

communicaticn 

deployment 

n4  <-  61,62,63; 

cl. gossip  -»  61,62  63  freq  5; 

nodes 

c2.send  ->  63  freq  10; 

nl: 

fixed 

c2.send  ->  61  freq  2; 

n2; 

n2  <-  61 

n3; 

n4; 

Figure  3:  Annotations  for  deployment  constrain' a 


The  fixed  section  lists  each  component  which  must  be  deployed  to  a  particular  node.  Once 
again,  a  statement  of  the  form  x  <-  y  means  that  component  y  must  be  issigned  to  host  x. 

Reliability  and  fault-tolerance  consideration  may  require  that  some  .groups  of  components  be 
separated  or  co-located.  For  instance,  data  replicas  should  be  hosted  on  different  nodes  while  tightly 
coupled  modules  (a  communication  channel  and  its  replica)  should  Lx  co-located  for  efficiency 
reasons.  The  separated  and  together  sections  can  be  used  to  specify  these  requirements  and  define 
lists  of  sets  of  modules.  In  our  example,  components  nl  and  a2  must  be  assigned  to  distinct  nodes, 
components  61,  62,  and  63  must  be  assigned  to  distinct  nodes,  and  compmients  «1  and  61  must  be 
assigned  to  the  same  node. 

Figure  4  illustrates  more  advanced  deployment  annotations.  The  first  of  these  is  the  constants 
section,  which  allows  the  user  to  name  literals3  used  within  the  specification.  Components  often 
will  “pass  through”  some  messages,  possibly  recording  information  from  the  messages  in  their  state. 
It  is  convenient  to  specify  these  common  message  frequencies  using  constants.  In  our  example,  / 1 
is  declared  to  have  the  value  5,  and  /2  is  declared  to  have  the  value  2.  Then  /I  and  /2  are  used 
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to  specify  the  relative  frequencies  of  the  three  transitions 

in  the  communication  section. 

automaton  C 

node  types 

equivalent 

components 

pc,  atm; 

{u3,  n4); 

al  :  A; 

«2  :  A  : 

nodes 

support 

61 :  D- 

nl  :  pc; 

nl  <-  al,a2; 

62:  R, 

n2; 

n2  <-  all; 

63  :  13; 

n3  :  sun; 

sun  <-  61,62,63; 

deployment 

n4  :  sun; 

communication 

constants 

connections 

al. gossip  ->  61,62,63  freq  /l; 

/ 1  :  Nat  :=5; 

{nl,n2}  bandwidth  25; 

a2.send  ->  63  freq  / 1  +  / 2  msgSize  4; 

/2  :  Nat  :=  2; 

{n2,  sun}; 

«2.$end  ->  61  freq  /2  msgSize  8; 

Figure  4:  More  advanced  deployment  annotations 


Node  types  represent  groups  of  nodes  with  the  same  capabilities.  Anywhere  a  group  appear  in  a 
specification,  the  node  type  may  be  used  instead.  The  node  types  of  a  deployment  must  be  declared 
with  the  keywords  node  types  followed  by  comma-separated  list  of  node  names  and  terminated  with 
a  semicolon.  Our  example  declares  two  node  types,  pc  and  sun.  Node  ul  is  of  node  type  pc ,  and 
nodes  n3  and  n4  are  members  of  sun  while  node  »2  has  no  node  type.  Whenever  pc  appears  in  the 
specification,  it  is  replaced  with  node  nl,  and  whenever  sun  is  used,  it  is  replaced  with  nodes  n3 
and  7*4.  For  instance,  the  connection  among  nodes  n2, 7i3,  and  rid  may  be  specified  as  {u2  sun). 

Some  groups  of  nodes  are  completely  equivalent,  in  that  they  support  the  same  set  of  compo¬ 
nents  and  are  connected  to  other  nodes  in  an  equivalent  manner.  Specifying  that  these  nodes  are 
equivalent  enables  the  opimizer  to  be  more  efficient.  The  sets  of  equivalent  nodes  are  listed  in  the 
equivalent  section.  In  our  example,  nodes  «3  and  rid  are  equivalent. 

In  some  applications  the  amount  of  data  transmitted  with  each  transition  is  essentially  the  same, 
but  in  other  applications  the  amount  of  data  transmitted  varies  from  one  transition  to  another  The 
deployment  annotations  allow  the  size  of  the  transmitted  data  to  be  specified  for  each  transition. 
The  optional  stanza  msgSize  expr  may  be  added  to  each  transition  listed  in  the  communication 
section.  Each  message  size  expression  is  interpreted  as  a  multiplicative  factor  of  an  unspecified  unit 
of  transmitted  data.  In  our  example  the  gossip  messages  from  component  ill  to  components  61, 
62,  and  63  are  of  size  1,  the  send  messages  from  component  «2  to  component  63  are  of  size  4,  and 
the  send  messages  from  component  «2  to  component  61  are  of  size  8. 

A  connection  may  have  a  bandwidth  limitation.  This  is  specified  by  appending  the  stanza 
bandwidth  expr  where  the  expression  specifies  the  maximum  bandwidth  for  the  set  of  nodes  in  the 
corresponding  connection.  The  bandwidth  expression  is  interpreted  as  the  maximum  amount  of 
data  which  may  pass  through  the  connection  during  a  time  period,  expressed  as  a  factor  of  a  unit 
of  transmitted  data.  The  implementation  assumes  that  each  transition  uses  a  single  path  for  data 
transmission.  In  our  example  the  connection  between  nodes  n  1  and  «2  lias  a  maximum  bandwidth 
of  25. 

5.2  Language  Extensions  for  Deployment  Annotations 

Deployment  annotations  are  added  to  the  Tempo  language  as  an  optional  extension  to  the  defini¬ 
tion  of  a  composite  automaton.  The  deployment  specification  begins  with  the  keyword  deployment 
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composed Aut  omat  on  components  hidden  ActionScfs?  comp  Schedule*,  deployment ? 

deployment deployment ’  constants?  nodeT'ypes ?  nodes 

connections  equivalent ?  constraint*  communication 

constants  ::=  'constants  1  (constant  *,  )+ 
eofisfan/ :s=lD  :  typcRcf  expr 

uodel)jpcs  ::=’node*  'types’  nodeType(,  uodeT'ypc)*  ; 
nodeTypc  ::=1D 

nodes  ’nodes *  (node  ;  )+ 

no</e::=ID([  varList  (,  varList  )*  ]  )?  (:  vodeType  )?  dcployWhcre ? 

cowice/jons  ’ connections’  ({  nodeS pec  List  }  (’bandwidth’  expr)?  ;  }  + 
equivalent  equivalent’  ({  nodeSpecList  }  i  )  + 

const inint  ::  =  support 
J  together 
|  separated 
j  Jixed 

support  ’support'  (tiodcSpcc  <  -  (’all’  |  compSpccList  }  ;  )  + 

/ojei/ier  together  ’  ({  compSpccList  }  ;  )  + 

separated  ::=  ’ separated'  ({  «wip6'pcc£ts/  }  ;  )-f* 
fixed  ::=  ’f  ixed'  (norfe/ruterice  <  -  cornp/ns/aricc  ;  )  + 

communtcarton  ’  conuounication’  commSpcc 

commSpcc  ’  for’  ID  'in’  INT  .  .  I  NT  ’do’  commSpcc  +  ’od’ 

|  commDxmsition 

COmmDansttion  ::=compTran$ition  -  >  compSpccList  ’freq’  expr  ( ’tasgSize’  expr)?  , 
comp?)nnsition  ::=conipInstancc  .  ID  ((  cxpr(,  expr)*  )  )? 

nodeSpecList  ::=nodeSpcc  ( ,  nod cS pec  )* 
nodeSpec  node  I  ns  lance  dcployWhcre? 

\  node  Type 

nodelnstancc  :;=1D  ( [  expr(,  expr)*  )  )? 

compSpccList  :•.=  comp  Spec  (,  compSpcc  )* 
compSpec  ::s= complnstance  deploy  Whcm? 
complnstancc  ::=ID  ([  expr(,  expr)*  )  )? 

dcployWhcre  ’where’  paramRangc  (A  paramRonge  )* 
paramllangc  :*.=ID  ’\in’  INT.  .  INT 

plainOp  ::=as  before  |  .  . 

expr  a— as  before  j  expr  ( .  ,  expr  )  + 

Figure  5:  EBNF  Grammar  fragment  for  deployment  expressions. 
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followed  by  optional  constants  and  node  types  specifications,  required  nodes  and  connections  spec¬ 
ifications,  optional  equivalent  nodes  and  constraint  specifications,  and  a  required  communication 
specification.  The  components  to  be  deployed  to  a  network  arc  the  component  parts  of  the  com¬ 
posite  automaton. 

The  constants  specification,  if  present  begins  with  the  keyword  constants  followed  by  one  or 
more  declarations  of  constant  variables.  Each  declaration  begins  with  an  identifier  corresponding  to 
the  name  of  the  variable,  followed  by  a  colon,  the  data  type  of  the  variable,  the  assignment  operator 

the  value  of  the  variable,  and  a  semicolon.  The  scope  of  a  constant  variable  declaration  is  the 
body  of  the  deployment  specification  As  the  name  implies,  the  value  of  a  constant  variable  cannot 
be  changed.  Constants  may  be  used  in  expressions  to  specify  bandwidth  limitations  of  connections 
and  frequencies  and  message  sizes  of  communicating  transitions,  for  example. 

The  node  types  specification,  if  present,  consists  of  the  keywords  node  types,  one  or  more 
identifiers,  separated  by  commas,  and  a  semicolon.  Each  identifier  is  the  name  of  a  node  type, 
which  is  just  a  shorthand  name  for  a  group  of  nodes.  A  node  may  belong  to  at  most  one  node  type. 

The  mandatory  nodes  specification  identifies  the  host  computer  nodes  onto  which  the  Tempo 
components  are  to  be  deployed.  It  begins  with  the  keyword  nodes  followed  by  one  or  more  node 
declarations,  each  ending  with  a  semicolon.  Eacii  node,  declaration  begins  with  an  identifier,  cor¬ 
responding  to  the  name  of  the  node,  and  a  list  of  its  parameters,  if  any,  separated  by  commas  and 
enclosed  in  square  brackets.  Each  parameter  specification  consists  of  an  identifier,  corresponding 
to  the  local  name  of  the  parameter,  followed  by  a  colon  and  its  data  type.  If  multiple,  adjacent 
parameters  arc  of  the  same  data  type,  their  identifiers  may  be  separated  by  commas  and  followed 
by  a  single  colon  and  their  common  data  type.  After  the  node  name  and  parameters,  there  is  an 
optional  node  type  designation  consisting  of  a  colon  and  the  identifier  of  the  node’s  type,  and  an 
optional  where  clause. 

A  node’s  where  clause  specifies  the  ranges  of  values  for  the  node’s  parameters.  It  begins  with 
the  keyword  where,  followed  by  one  or  more  parameter  range  specifications,  separated  by  the  AND 
operator  /\,  and  ends  with  a  semicolon.  Each  identifier  used  within  the  node’s  parameter  specifi¬ 
cations  must  have  a  corresponding  parameter  range  specification  in  the  where  clause  consisting  of 
the  identifier,  the  keyword  \in,  and  the  integer  lower  and  upper  bounds  for  the  identifier’s  values, 
separated  by  two  periods  (.  .). 

The  mandatory  connection  section  itemizes  the  hardware  communication  links  in  the  network, 
be  they  simple  communication  cables  connecting  two  nodes  or  Ethernet  cables  or  switches  connect¬ 
ing  multiple  nodes.  The  section  begins  with  the  keyword  connections  and  contains,  for  each  link, 
the  list  of  directly  connected  nodes,  separated  by  commas,  enclosed  in  braces,  and  terminated  with 
a  semicolon.  If  a  link  has  limited  bandwidth,  that  is  specified  after  the  closing  brace  but  before 
the  terminating  semicolon,  with  the  keyword  bandwidth  followed  by  a  measure  of  the  limited  ca¬ 
pacity.  If  no  bandwidth  is  specified  for  a  link,  it  is  assnmed  that  the  bandwidth  of  the  connection 
is  sufficient  to  be  considered  unlimited  for  the  purposes  of  deployment. 

For  each  connection,  each  node  specification  consists  of  an  identifier,  corresponding  to  the  name 
of  the  node,  and  a  fist  of  its  parameters,  if  any,  separated  by  commas  and  enclosed  in  square  brackets. 
An  optional  where  clause  may  be.  used  to  refer  to  a  group  of  nodes  where  each  identifier  used  within 
the  node’s  parameter  specifications  must  have  a  corresponding  parameter  range  specification  in  the 
where  clause,  as  above.  A  node  type  identifier  also  may  be  used  to  refer  to  a  group  of  nodes  for  a 
connection,  if  all  the  nodes  of  that  type  are  connected  with  a  single  communication  fink. 

Equivalent  nodes,  if  any,  are  listed  next,  beginning  witii  the  keyword  equivalent  followed  by 


VeroModo,  Inc. 


Final  Technical  Report 


FA955U-07-C-01U 


-24- 


eacli  group  of  equivalent  nodes.  Within  each  group  the  individual  uod-is  or  groups  of  nodes  are 
specified  in  the  same  maimer  as  for  connections,  with  the  node  specifications  separated  by  commas, 
enclosed  in  braces,  and  terminated  with  a  semicolon.  Providing  the  sets  of  equivalent  nodes  enables 
the  optimal  deployment  tu  he  calculated  more  efficiently. 

Several  constraints  may  be  placed  on  the  deployment  of  components  to  nodes.  For  example, 
some  components  may  execute  only  on  a  subset  of  the  network's  nodes  Some  components  must 
be  deployed  to  the  same  node  while  other  components  must  not  be  cj-located.  Filially,  some 
cumponents  must  be  deployed  to  particular  nodes. 

Each  component  specification  consists  of  an  identifier,  corresponding  >  j  the  name  of  the  compo¬ 
nent,  and  a  list  of  its  parameters,  if  any,  separated  by  commas  and  enclosed  in  square  brackets.  An 
optional  where  clause  may  be  used  to  refer  to  a  group  of  components,  where  each  unbound  identifier 
used  within  the  component’s  parameter  specifications  must  have  a  corresponding  parameter  range 
specification  in  the  where  clause,  as  for  nodes. 

The  support  constraints  if  present  specify  which  components  may  bt  deployed  to  which  nodes. 
The  section  begins  with  the  keyword  support  and  gives  for  each  nod?  or  group  of  nodes  the 
list  of  components  they  support.  Each  individual  support  constraint  *egins  with  an  identifier, 
corresponding  to  the  name  of  a  node  or  node  type.  If  the  identifier  corresponds  to  the  name  of  a 
node,  it  is  followed  by  the  list  of  the  node’s  parameters,  if  any,  separatee  by  commas  and  enclosed 
in  square  brackets,  and  an  optional  where  clause  specifying  the  range  of  values  for  the  node’s 
parameters.  The  node  specification  is  followed  by  the  symbol  <-  and  either  a  list  of  specifications 
for  the  supported  components,  separated  by  commas,  or  the  keyword  al  L  if  the  nodes  support  all 
components.  Each  support  constraint  ends  in  a  semicolon.  If  no  suppon  constraints  are  included 
in  a  deployment  specification,  every  component  may  run  on  every  nc  3c.;  otherwise,  a  support 
constraint  must  be  supplied  for  each  node. 

The  together  constraints,  if  present,  specify  groups  of  components  that  must  be  deployed  to¬ 
gether  to  the  same  nodes.  The  section  begins  with  the  keyword  together  and  consists  of  groups 
of  component  specifications,  separated  by  commas,  enclosed  in  braces.  ?  id  terminated  with  semi¬ 
colons. 

Similarly,  the  separated  constraints,  if  present,  specify  groups  of  c  Dinpoiienls  that  must  be 
deployed  to  separate  distinct  nodes.  The  section  begins  with  the  keywor  3  separated  and  consists 
of  groups  of  component  specifications,  separated  by  commas,  enclosed  1  braces,  and  terminated 
with  semicolons. 

The  fixed  constraints,  if  present,  identify  the  components  that  must  jc  deployed  to  particular 
nodes.  The  section  begins  with  the  keyword  fixed  Each  individual  fixed  constraint  begins  with 
an  identifier,  corresponding  to  the  name  of  the  node  onto  which  the  component  is  to  be  deployed, 
and  a  list  of  its  parameters,  if  any,  separated  by  commas  and  enclosed  i:  square  brackets.  This  is 
followed  by  the  symbol  <-  and  a  second  identifier,  corresponding  to  tint  name  of  the  component, 
and  a  list  of  its  parameters,  if  any,  separated  by  commas  and  enclosed  in  square  brackets.  Each 
constraint  ends  with  a  semicolon.  Since  a  fixed  constraint  assigns  a  single  component  to  a  single 
node,  neither  a  where  clause  nor  a  node  type  may  be  used  in  the  specifi  rations. 

The  final  deployment  section,  a  mandatory  communication  section,  ;•  lecilies  the  frequencies  of 
the  composite  automaton’s  communicating  transitions.  It  consists  of  th*  keyword  communication 
followed  by  the  individual  transition  specifications.  Each  transition  specification  begins  with  an 
identifier,  corresponding  to  the  name  of  the  “sending”  component,  a  lis  of  its  parameters,  if  any, 
separated  by  commas  and  enclosed  in  square  brackets,  followed  by  the  dot  symbol  . ,  and  a  second 
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identifier,  corresponding  to  the  name  of  one  of  the  component’s  transitions.  These  are  followed  by 
the  symbol  ->  and  a  list  of  component  specifications,  separated  by  commas,  for  the  “receiving* 
components.  The  transition  specification  ends  with  the  keyword  freq  followed  by  an  expression  for 
the  frequency  of  the  transition,  optionally  the  keyword  msgSize  and  an  expression  for  the  average 
size  of  the  transition’s  “message”,  and  a  semicolon.  The  transition  must  be  an  output  transition 
of  the  “sending"  component  and  an  input  transition  of  each  of  the  “receiving”  components.  The 
units  and  time  interval  for  the  transition  frequency  and  message  size  expressions  arc  not  specified 
as  part  of  the  deployment  annotations;  it  is  assumed  that  application-specific  units  arc  selected 
and  uniformly  used  for  all  transition  specifications  and  connection  bandwidth  limitations 

A  for  loop  can  be  used  to  specify  groups  of  similar  transitions,  such  as  those  of  gossiping  data 
replicas.  The  for  loop  begins  with  the  keyword  for,  an  identifier  for  the  loop  variable,  the  keyword 
in,  integers  for  the  lower  and  upper  bounds  on  the  loop  variable,  separated  by  the  symbol  . ., 
and  the  keyword  do.  These  are  followed  by  one  or  more  transition  specifications,  as  above,  and 
the  keyword  od.  Each  occurrence  of  the  loop  variable  among  the  parameters  of  the  “sending”  and 
“receiving”  component  specifications  is  replaced,  in  turn,  by  each  value  between  the  loop  variable’s 
bounds  (inclusively). 

5.3  Eventually  Serializable  Data  Service  Annotations 

An  Eventually-Serializable  Data  Service  (ESDS)  [8,  7]  maintains  multiple  copies  of  its  data  for  fault 
tolerance,  but  it  selectively  relaxes  the  consistency  requirements  among  its  copies  of  the  data  in 
exchange  for  improved  performance.  ESDS  guarantees  that  the  replicated  data  will  eventually  be 
consistent,  although  it  may  not  be  at  a  particular  point  during  the  execution. 

ESDS  consists  of  four  types  of  components:  clients,  front  ends,  replicas,  and  channels.  The 
clients  request  operations  to  be  performed  on  the  shared  data  and  receive  responses  containing  the 
results  of  these  operations.  The  front  ends  communicate  with  the  clients,  keeping  track  of  all  their 
pending  requests  and  forwarding  those  requests  to  one  or  more  of  the  replicas.  Each  replica  keeps 
a  copy  of  the  requested  operations  on  the  shared  data  and  a  partial  order  on  those  operations; 
the  partial  order  must  be  consistent  with  both  the  responses  and  the  eventual  total  ordering  of 
the  operations.  The  front  ends  do  not  send  every  request  to  every  replica,  so  the  replicas  “gossip” 
among  themselves  to  stay  informed  about  all  the  operations  that  have  been  received  and  processed. 
The  channels  are  used  to  transmit  these  gossip  messages. 

Figures  C  and  7  illustrate  the  component  communication  of  an  example  ESDS  and  the  computer 
network  onto  which  it  is  to  be  deployed.  This  example  first  appeared  in  [Ij  More  recently, 
the  example  was  hand-coded  in  Comet  to  test  the  feasibility  of  using  constraint  programming  to 
determine  optimal  deployments  [21 J.  Figure  8  contains*  the  Tempo  deployment  annotations  for  this 
example. 

The  example  consists  of  four  clients,  c[l],  c[2],  e[3],  and  c[4],  two  front  ends,  /e[l]  and  /c[2],  and 
six  replicas,  r[l],  r [2],  r[3],  r[4),  r [5],  and  r[6).  Clients  c[l]  and  c[2]  make  their  requests  of  /e[l],  and 
clients  c[3]  and  c[4]  make  their  requests  of  /c[2].  Front  end  /e[l],  in  turn,  forwards  its  requests  to 
r[I],  and  front  end  /e[2]  forwards  its  requests  to  r[4|.  The  components  are  to  be  deployed  to  four 
PCs,  pc[l].  pc(2],  pc[3],  and  pc[4],  and  ten  Sun  servers,  stm[l]  through  suu[10].  Each  of  the  PCs  is 
connected  to  a  Sun,  and  all  of  the  Suns  are  connected  to  each  other  with  a  common  connector. 

Several  additional  requirements  are  placed  on  the  deployment.  First,  c[l],  c(2],  and  c[3]  must 
be  deployed  to  PCs;  the  rest  of  the  components  must  be  deployed  to  Suns.  Second,  to  maintain 
fault  tolerance,  the  replicas  must  be  deployed  to  distinct  computers.  Third,  /e[l]  must  be  deployed 
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Figure  C;  Components  for  ESDS  example. 


to  stm[2],  and  /e(2)  must  be  deployed  to  sun (3).  This  last  requirement  was  added  to  make  the 
deployment  optimization  more  tractable  in  its  initial  implementation  [l] 

The  Tempo  implementation  of  ESDS  requires  channels  between  each  pair  of  replicas,  making 
the  model  more  consistent  with  the  original  ESDS  model  [8]  These  channels  are  named  c/i(I,l| 
through  c/i[G,G|  where  replica  r(i]  uses  channel  di\ij\  to  gossip  with  r:plica  r(jj.  The  channels 
require  an  additional  set  of  deployment  constraints,  namely,  each  replica  r(i]  must  be  co-localed 
with  each  of  its  channels  c/i(i,j]. 

The  ESDS  automaton  in  Figure  8  stores  both  its  components  and  its  nodes  in  arrays.  For 
example,  the  Client  components  are  declared  in  the  components  section  with 

c[i :  Nat]  :  Clie7il{i)  where  i  \in  1..4; 

Note  that  the  data  type  of  the  array  index  must  be  declared  as  an  Nat  The  range  of  the  array 
indices  is  specified  with  the  keyword  where  followed  by  the  index  variable,  the  keyword  \in,  the 
lower  bound  of  the  indices,  two  periods,  and  the  upper  bound  oT  the  indices.  Array  indices  need 
not  start  with  1.  Both  the  component  and  node  arrays  may  be  multi-dime  isional  such  as  the  array 
c/i  of  Channel  components. 

The  components  and  nodes  that  are  stored  in  arrays  may  be  accessed  both  individually,  such  as 
pc(3]  or  as  a  group  of  sequential  elements,  such  as  svn\i\  where  i  \in  5..1C  in  the  equivalent  section. 
Again,  a  where  clause  is  used  to  specify  the  range  of  array  indices.  Note  that  the  range  of  indices 
may  be  used  to  specify  a  subset  of  the  elements  in  an  array. 

In  the  communications  section  nested  for  loops  may  be  used  to  spec  Ty  similar  transitions  for 
arrays  of  components.  This  is  particularly  helpful  in  the  ESDS  automato  for  specifying  the  gossip 
frequencies  of  the  72  transitions  among  the  replicas  and  channels. 

6  Generating  deployment  models 

The  deployment  annotations  are  incorporated  into  the  Tempo  Toolkit  [F7]  as  a  new  plug-in.  The 
plug-in  translates  the  annotations  into  a  Comet  constraint  program,  wind  is  subsequently  executed 
to  determine  an  optimal  allocation  of  components  to  computing  nodes  in  the  target  network.  We 
now  describe  in  detail  the  translation  scheme,  the  resulting  Comet  program,  and  the  language 
restrictions  designed  to  enable  effective  automatic  generation  of  optimal  *cployinent. 
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automaton  ESDS 

fixed 

components 

stm[2]  <-  /c(l]; 

c[i :  Nat]  :  Cfietif(t)  where  i  \in  1..4; 

sun(3]  <-  /c[2]; 

fe[i.  Nat]  :  Front End[i)  where  i  \in  1..2; 

separated 

r[i :  Nat]  :  Ilcplica(i)  where  i  \in  1..C; 

{r[i]  where  i  \in  1..G}; 

c[i :  Nat.  j  :  Nat]  :  Channd(i,  j)  where 

together 

i  \in  1..6  /\  j  \in  1..G; 

{r[l],  c/i(l,  j]  where  j  \in  1..6}; 

deployment 

{r[2] r  c/*[2,  j]  where  j  \in  1..6}; 

constants 

{r|3]  c/i[3,  j]  where  j  \in  1..6}; 

clFrcq  :  Nat  :=  10; 

{r[4j  c/j [4,  j]  where  j  \in  1..G); 

c2Freq  :  Nat  :=  2; 

{r[5],  c/i[5,  j]  where  j  \in  1..6}; 

cSFreq  Nat  :=■  10; 

{r(G],c/i[G,  j]  where  j  \in,l..G}; 

c4Freq  :  Nat  :=  5; 

gossipFreq  :  Nat  :=  5; 

communication 

nodes 

c[l]. request  ->  /e(l]  freq  cl Freq\ 

pcji  ;  Nat]  where  i  \in  1..4; 

c[2]. request  ->  /e[lj  freq  c2Freq\ 

sun[i  :  Nat]  where  i  \in  1..  10; 

c(3]. request  ->  /e[2]  freq  c3Freq\ 

connections 

c(4). request  ->  /c[2]  freq  c4Freq; 

{pc(l]>*«n[l]}; 

/e(l].send  ->  r[l]  freq  clFreq  +  c2Freq\ 

{pc[2],sun[2]}; 

/ejlj. response  ->  c(l]  freq  clFreg; 

{;>c[3],  stm[3j}; 

/ejlj.  response  ->  c(2]  freq  c2Freq\ 

{pc[4],  sun|4]}; 

/e[2].send  ->  r[4]  freq  cSFreq  +c4 Frcq\ 

{sun(i]  where  i  \in  1..10}; 

Jc[2). response  ->  c{3]  freq  c3Frcqm, 

equivalent 

fe.\2]. response  ->  c[4]  freq  c4 Freq-, 

(sun[i]  where  i  \in  1.  4); 

r[l]  receive  ->  /e[lj  freq  cl  Freq  +  c2Freq\ 

{sunjij  where  i  \in  5  . 10}; 

r[4]. receive  ->  /c[2]  freq  c3Frcq  +  c4Freq; 

support 

for  i  in  1..G  do 

pc|i]  where  i  \in  1..4  <-  c[l],c[2],c[3]; 

for  j  in  1..6  do 

«un[i]  where  i  \in  1..10  <-  c[4], /e(l],/e[2], 

r[t], gossipSend  ->  ch[i,  j]  freq  gossipFreq ; 

r[i]  where  i  \in  1..G; 

ch [i,  j) .gossipReceive  ->  r\j]  freq  gossipFreq ; 

5un(t]  where  i  \in  1..10  <-  c[*f  j]  where 

od 

t  \in  1..G  /\  j  \in  1..G; 

od 

Figure  8:  Deployment  annotations  for  ESDS  example 
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G.l  Translation  Scheme 

The  deployment  annotations  are  extensions  to  the  Tempo  language., 
so  care  was  taken  to  minimize  their  impact  on  existing  Tempo  pro¬ 
grams.  To  that  end,  the  deployment  annotations  only  occur  within 
the  definition  of  composite  automata,  and  they  are  isolated  within 
those  definitions  to  a  separate  new  section  beginning  with  the  key¬ 
word  deployment 

The  translation  process  begins  by  enumerating  all  the  components 
and  nodes  and  assigning  their  names,  as  provided  by  the  Tempo  mod¬ 
eler,  to  two  arrays  of  type  string.  The  Comet  program  then  identifies 
the  components  and  nodes  by  the  indices  of  their  names  in  these  ar¬ 
rays.  For  example,  for  the  deployment  annotations  in  Figure  2  the 
array  of  node  names  is  [”ul” , ”u2” ,  ”n3” ,  ”  n.4" J  and  the  connection 
sets  {ul,n2}  and  {n2,»3,n4}  are  encoded  as  {0.1}  and  {1,2,3}.  At 
the  end  of  execution,  the  Comet  program  displays  the  optimal  deploy¬ 
ment  with  the  Tempo  modeler’s  names.  Figure  9  shows  the  resulting 
deployment  output  for  all  but  the  channel  components  of  the  ESDS  Figure  9:  Deployment  for 
example  in  Section  5.3.  ESDS  Example 

The  variables  declared  in  a  constants  section  of  the  deployment  specification  arc  carried  over  to 
the  Comet  program  and  declared  and  initialized  there.  When  these  variab  es  are  subsequently  used 
to  specify  communication  frequencies,  for  example,  the  variable  names,  rather  than  their  values,  are 
encoded  in  the  Comet  program  This  allows  arbitrary  arithmetic  expressions  for  communication 
frequencies  without  requiring  the  Tempo  front-end  to  evaluate  those  expressions. 

G.2  Comet  Program 

The  output  of  the  translation  stage  is  a  Comet  program.  That  program  relies  on  Constraint 
Programming  technology'  to  solve  the  deployment  problem  optimally.  Constraint  programming 
delivers  a  complete  solution  method.  Constraint  programs  revolve  around  two  components.  A 
declarative  component  state  the  discrete  decision  variables,  the  constra  nts  that  every  solutions 
must  satisfy  and  the  objective  function.  The  second  component  focuses  on  the  specification  of  a 
tree-search  process  revolving  around  an  implicit  enumeration. 

The  Tempo  translator  for  Comet  produces  a  complete  model  that  features  both  the  declarative 
component  and  an  instantiation  of  a  search  template.  That  template  takes  advantage  of  the  prop¬ 
erties  conveyed  through  the  annotations  such  as  the  equivalence  classes  (specified  in  the  equivalent 
section)  among  nodes  to  implement  a  symmetry  breaking  procedure  lhalconsiderably  reduces  the 
running  lime. 

As  with  equivalent  and  support,  the  generated  Comet  code  varies  depending  upon  whether  or 
not  bandwidth  constraints  arc  included  in  the  deployment  specification.  Five  different  interpreta¬ 
tions  of  the  bandwidth  constraints  were  considered. 

•  A  single  path  is  used  between  each  pair  of  nodes. 

•  A  single  path  is  used  between  each  pair  of  components. 

•  A  single  path  is  used  for  each  transition. 


Deployment : 
pc [2]  <-  c[l] 
pc  [2]  <-  c[2] 
pc  [3]  <-  c[3) 
sun[3]  <-  c[4] 
sun  [2]  <-  f e  [5] 
sun [3)  <-  fe[6] 
sun[2]  <-  r[7] 
sun [5]  <-  r[8] 
sun [1]  <-  r[9] 
sun [3]  <-  r[10] 
sun[4j  <-  r[ll] 
sun[6]  <-  r [ 123 
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•  A  single  path  is  used  for  each  message 

•  Multiple  paths  may  be  used  for  a  single  message. 

We  chose  to  use  a  single  path  between  each  pair  of  components  since  that  option  most  closely 
embodies  the  concept  of  establishing  a  connection  between  components.  Subsequent  versions  of 
the  deployment  plug-in  may  include  other  types  of  bandwidth  constraints,  or  even  include  commu¬ 
nication  load  balancing  among  the  connections. 

Performance-wise,  the  programs  generated  with  the  help  of  the  Tempo  translator  are  more  than 
competitive  with  hand-written  programs.  When  applied  to  the  ESDS  deployment,  the  generated 
program  is,  to  this  date.  the.  most  effective  way  to  solve  the  problem.  The  effectiveness  of  this 
approach,  when  compared  with  modern  mixed-integer  programming  solvers.,  is  reported  in  [22].  For 
the  ESDS  example  in  Section  5.3.  the  Comet  program  finds  the  optimal  deployme.nl  approximately 
20  times  faster  than  CPLEX  version  11  and  25,000  times  faster  than  the  hand-coded  C  program 
reported  in  [1]. 

6.3  Tempo  Language  Restrictions 

Each  of  the  Tempo  Toolkit  plug-ins  place  some  restrictions  on  the  Tempo  language  constructs 
which  are  supported,  and  the  deployment  plug-in  is  no  exception.  First,  since  the  components 
and  nodes  must  be  enumerable  the  contents  of  their  where  clauses  currently  are  limited  to  range 
sets  of  type  Nat  and  the  /\  operator,  as  in  c[i,  j)  where  i  \in  1..6  /\  j  \in  1..6.  Second,  nested 
composite  automata  are  not  supported,  pending  identification  of  distributed  systems  that  require 
this  modeling  complexity. 

Tempo  specifies  the  communication  among  components  implicitly;  each  output  transition  is 
linked  with  all  input  transitions  having  the  same  name  and  matching  parameters  in  other  com¬ 
ponents.  One  of  an  output  transition  s  parameters  often  specifies  an  identifier  for  the  component 
with  the  matching  input  transition.  This  is  particularly  useful  for  applications  using  arrays  of 
components. 

Unfortunately,  this  implicit  linking  through  parameter  values  makes  it  extremely  difficult  for 
the  Tempo  deployment  annotations  to  match  output  transitions  with  input  transitions  at  compile- 
time  as  needed,  rather  than  run- time.  The  current  annotations  use  explicit,  rather  than  implicit, 
transition  matching  as  a  result.  For  example  from  Figure  2,  a2,send  ->  63  freq  10;  gives  the 
frequency  of  the  send  output  transition  of  «2  when  it  is  linked  with  the  send  input  transition  of 
63,  and  ol. gossip  ->  61,62,63  freq  5;  gives  the  frequency  of  the  gossip  output  transition  of  «1  when 
it  is  linked  with  the  gossip  input  transitions  of  61,  62,  and  63.  The  downside  of  this  approach  is 
that  the  Tempo  front  end  can  only  do  limited  error  checking.  In  the  first  example,  the  front  end 
ensures  that  «2  and  63  have  send  transitions  of  the  proper  type,  but  it  does  not  ensure  that  the 
transitions  actually  will  link  in  a  run-time  setting  nor  does  it  ensure  that  there  aren’t  additional 
send  input  transitions  in  other  components  which  also  will  link  with  the  output  transition.  An 
alternate  communication  syntax  being  considered  is  u2.send(„,  3)  freq  10;,  which  implicitly  links 
the  parameter  3  to  a  parameter  of  type  const  in  the  send  input  transition  of  component  63. 

7  Solving  deployment  models 

This  section  describes  the  optimization  model  that  one  obtains  from  the  Tempo  translator  when 
it  is  applied  to  the  Eventually  Serializable  Data  Service  application.  The  section  starts  with  a  pre- 
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sentatiou  of  the  abstract  model  followed  by  its  incarnation  in  COMET  as  constraint  programming 
model. 

7.1  The  Abstract  Model 
The  input  data  consists  of 

•  The  set  of  software  modules  C. 

•  The  set  of  hosts  N. 

•  For  each  component,  the  subset  of  hosts  to  which  it  can  be  assigned.  In  the  following.  sCt„  is 
a  boolean  variable  equal  to  true  if  and  only  if  component  c  can  be  assigned  to  host  n. 

•  The  network  cost  is  directly  derived  from  its  topology  and  express: d  with  a  matrix  h  where 
/i,j  is  the  minimum  number  of  hops  required  to  send  a  message  fr:in  host  t  to  host  j.  Note 
that  hit  =  0  (local  messages  are  free). 

•  The  message  volumes.  In  the  following.  denotes  the  average  frequency  of  messages  sent 
from  component  a  to  component  b. 

•  The  separation  set  Sep  which  spccilies  that  the  components  in  each  S  6  Sep  must  be  hosted 
on  a  different  servers; 

•  The  co-location  set  Col  which  spediies  that  the  components  in  eac  S  6  Col  must  be  hosted 
on  the  same  servers; 

The  decision  variables  xc  are  associated  with  each  module  c  6  C  and  rc  =  n  if  component  c  is 
deployed  on  host  n.  An  optimal  deployment  minimizes 

aec  tec 

subject  to  the  following  constraints.  Each  component  may  only  be  assigned  to  a  host  that  supports 
it 

Vc  6  C  :  xc  6  {i  €  N  f  sCii  =  1}. 

For  each  separation  constraint  S  6  Sep ,  we  impose 

Vi,  j  €  S  :  i  j  Xi  Xj. 

Finally,  for  each  co-location  constraint  expressed  over  a  subset  of  components  S  6  Col ,  we  impose 

Vi,  j  e  S  :  x,  =  Xj. 


7.2  The  CP  Model 

The  Comet  constraint  program  generated  by  Tempo  for  the  Eventually  Serializable  Data  Service 
Deployment  Problem  is  shown  in  Figure  10.  We  review  its  main  compoi  ents. 
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i  range  Caps  —  1  nbC ap; 

3  range  Colors  =•  1  ..nbCotors; 

3  range  Orders  l..n  bOrders; 

4  range  Slabs  =  L.nbSlabs; 
s  int  capacities  [Caps]  = 

n  int  weight (Orders)  — 

7  int  color  (Orders)  =...; 

•j  set  {int}  colorOrders(c  in  Colors)  =  fillor(o  in  Orders)  (colorjo]  ==  c); 

10 

ii  int  maxCap  =  max(i  in  Caps)  csipacities(i); 

■2  int  1oss[c  in  0..inaxCap]  =  min(i  in  Caps:  capar.iiics(i]  >=  c)  capacities(i)  —  c; 
is 

m  Solver<Cl>>  m(); 
is  var<CP>{int}  x[0rdersj(m>S1abs); 
ir>  var<CP>{inl}  l(Slabs](m,0.. inaxCap); 

is  miniinize<in>  sum(s  in  Slabs)  loss[l(s]j 

19  subject  to  { 

20  m. post  (mull  ikii5ip.«!ack(xtweight,l)); 

71  forall(s  in  Slabs) 

22  m.posi  (sum(c  in  Colors)  (or(o  in  coloiOrders(ej)  (x[o)  ==  s))  <=  2); 

23  }  using  { 

24  forall(o  in  Orders)  by  (x(o].getSize(),-wciglit[oj)  { 

25  int  ms  =  max(0,maxBound(x)); 

20  tryali<in>(s  In  Slabs:  s  <=  ms  +  1) 

27  in.labcl(x[o],s); 

28  on  Failure 

79  m.diff(x[o],s); 

30  } 

31  } 


Figure  10:  The  Constraint-Programming  Model  in  COMET 


7.2.1  *  The  Model 

The  model  is  depicted  in  lines  1-21  in  Figure  10.  The  data  declarations  arc  specified  in  lines  2-10 
and  should  be  self-explanatory.  The  decision  variables  are  declared  in  line  10  (they  are  the  same 
as  in  the  ESDS  model  given  earlier):  variable  x(cj  specifies  the  host  of  component  c  and  its  domain 
is  computed  from  tlie  support  matrix  s. 

The  objective  function  is  specified  in  lines  12-13  and  eliminates  the  diagonal  elements  (since 
/ij  ,  =  0  for  every  i  G  Ar).  The  CP  formulation  features  a  two-dimensional  element  constraint  since 
the  matrix  h  is  indexed  by  variables.  Lines  15-18  state  the  co-location  constraints:  for  each  set  5 
(line  15),  an  element  ci  6  5  is  selected  (randomly)  and  the  model  imposes  the  constraint  zC)  =  xn 
for  each  other  elements  c-i  in  S.  Lines  19-20  state  the  separation  constraints  for  every  set  in  Sep 
using  alldifierent  constraints.  The  onDomains  annotations  indicate  that  arc-consistency  must  be 
enforced  on  tlu;  equations  and  alldifTciciil  constraints. 

It  is  interesting  to  discuss  the  pruning  performed  by  the  objective  function  when  an  upper  bound 
is  available.  In  COMET,  the  multi-dimensional  element  constraints  are  implemented  in  terms  of  a 
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table  T  which  contains  all  the  tuples 


(a.  6,  lia,b)  (a.  6  €  C). 


COMET  also  creates  a  new  variable  au^  for  each  term  hXa ^  in  the  objective  and  imposes  the 
constraint 


(xa,  xb,caib)  6  T 

on  which  it  achieves  arc  consistency.  With  this  in  place,  the  objective  tlen  becomes 


«eC6ec 


7.2.2  The  Search  Procedure 

The  search  procedure  is  depicted  in  lines  23  29  It  is  a  variable  labeling  with  dynamic  variable  and 
value  orderings.  Lines  24-28  are  iterated  until  all  variables  are  bound  (line  23)  and  each  iteration 
uoudeterininistically  assigns  a  variable  x[i]  to  a  host  n  (lines  25-26). 

It  is  interesting  to  review  the  variable  and  value  orderings  which  are  motivated  by  the  structure 
of  the  objective  function 


«eC  jec 


In  the  objective,  the  largest  contributions  are  induced  by  assignments  o.'  components  i  and  j  that 
are  communicating  heavily  and  are  placed  on  distant  hosts.  As  a  result,  the  variable  and  value 
ordering  are  based  on  two  ideas: 

1*  Assign  first  a  component  i  whose  communication  frequency  with  a  component  j  is 

maximal  (line  24); 

2.  Tty  the  hosts  for  component  i  in  increasing  number  of  hops  requ  red  to  communicate  with 
component  j  (line  25). 

The  variable  selection  thus  selects  first  components  with  the  heaviest  (single)  communications, 
while  the  value  selection  tries  to  deploy  the  components  to  minimize  the  number  of  hops. 

Arc-Consistency  for  fitering  The  CP  model  used  here  is  quite  elegant  since  it  enforces  arc 
consistency  on  all  constraints  and  the  objective  function.  One  may  wonder  whether  arc  consistency 
is  critical  in  ESDSDPs  or  whether  a  weaker  form  of  consistency  is  sufficient.  Table  1  depicts  a 
comparison  of  a  bound-consistency  model  and  ail  arc-consistency  model  oil  a  collection  of  synthetic 
benchmarks.  The  second  and  the  third  column  report  the  results  of  the  CP  solver  when  bound 
consistency  is  enforced  on  the  objective,  while  the  fourth  and  the  fifth  columns  report  the  perfor¬ 
mance  for  the  arc-consistency  model.  The  experimental  results  show  a  dramatic  loss  in  performance 
when  arc  consistency  is  not  used  and  underline  the  importance  of  using  sophisticated  constraint 
programming  techniques  to  deliver  the  desired  performances. 
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Algo 

CP-BC 

CP-AC 

Bench 

To  d 

#CHPT 

Tnd 

#CHPT 

S1MPLE2 

1.20 

7582 

0.23 

2510 

SIMPLE1 

0.11 

46874 

1.38 

15408 

SIMPLEO 

37.21 

307365 

7.75 

87491 

fe3c5pc 

94.81 

748118 

2.76 

14597 

fe3c5pc5 

039.87 

4705378 

4.04 

24130 

fe3c5sun 

100.39 

1330353 

0.29 

30621 

fe3c0pc5 

1039.20 

7238005 

3.54 

18547 

fe3c7pc5 

2107.10 

14446831 

7.83 

35726 

fe3c7pc5CS 

1910.50 

12940789 

7.77 

35312 

fe3c7pc5CST 

1280.37 

8557292 

13.08 

70495 

fe3dist 

93.04 

839781 

4.16 

29750 

SCSSlSNUFE 

02.80 

482601 

43.34 

392628 

SCSS2SNUFE 

00.47 

442373 

06.43 

380117 

SCSS2SNCFE 

30.41 

246228 

50.83 

322472 

HYPER8 

7053.00 

33028203 

65.07 

123213 

HYPER16 

34570.90 

150832040 

237.53 

513051 

Table  1  The  Value  of  Arc  Consistency  for  the  CP  Model 

Exploiting  Value  Symmetries  As  discussed  earlier,  some  instances  of  the  ESDS  deployment 
problem  feature  a  variety  of  symmetries,  which  can  be  removed  to  improve  the  search  performance 
without  sacrificing  optimality  guarantees.  Techniques  for  removing  these  symmetries  during  search 
are  well-known  (see,  for  instance,  [33]) 

Figure  11  illustrates  how  to  enhance  the  search  procedures  presented  earlier  with  symmetry 
breaking.  The  sets  of  equivalent  hosts  are  supplied  as  addit  ional  input  data  and  are  used  to  deter¬ 
mine  the  set  of  non-equivalent  hosts  (lines  12).  Each  iteration  of  the  search  procedure  calculates 
the  set  of  nodes  that  are  bound  in  line  6  and  the  set  of  nodes  that  are  eligible  to  host  the  next 
component  with  lines  7  through  11.  Line  7  starts  by  initializing  the  searchNodcs ,  to  all  the  non¬ 
equivalent  nodes  plus  all  the  nodes  on  which  components  are  already  deployed.  The  loop  in  lines 
8-11  simply  adds  to  seurchNodes  one  still  unused  node  from  each  equivalence  class. 

8  A  Formal  Treatment  of  an  Abstract  Channel  Implementation 
Using  Java  Sockets  and  TCP 

Our  earlier  research  substantiates  our  ability  to  implement  practical  techniques  for  generating  dis¬ 
tributed  code,  automatically,  starting  from  formal  Inpnt/Ontput  Automata  (IOA)  specifications  in 
Tempo.  Namely  we  have  developed  an  automated  code-generator  for  IOA  programs  in  a  specific 
node-channel  form  that  produces  Java  code  miming  over  MPI  on  a  local  area  network  [30,  31),  and 
have  used  this  to  generate  running  versions  of  a  variety  of  basic  distributed  algorithms  [10],  We  have 
also  developed  two  complete  distributed  systems  by  manually  (but  systematically)  translating  for¬ 
mal  IOA  specifications  to  distributed  code,  using  C++/MPI  to  implement  an  evcntually-scrializablo. 
data  service  [7),  and  using  Java/sockets  to  implement  a  reconfigurable  atomic  read/wrile  memory 
service,  called  Rauibo,  e.g.  see  [20,  1 1 J  The  methodology  that  emerged  as  a  part  of  the  develop- 
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i  set{set{int}}  Eq  -  //The  equivalent  host  nets 
7  set{int}  NolEq  -  ...;  //The  non- equivalent  hosts 
s 

t  while  (sum(k  in  C)  x{k].bound()  <  C.gclSixc())  { 

5  se!ectMax(a  In  C:  !x[a].bou»d(),  b  in  C)(f[a,b])  { 

n  setfint}  bound  Nodes  —  collects  in  C  :  x[s].boi*nd())  x[s]; 

7  setfint}  scardiNortes  —  notEq  union(union(c  in  Eq)  e  inter  bound  Nodes); 

-  for  ail  (c  in  Eq)  { 

o  set{int}  fen  =■  c  \  bound  Nodes: 

in  if  (card  (fen)  >  0)  search  Nodes,  insert  (min  (ii  in  fen)  n); 

ii  } 

19  int  k  —  inin(k  hi  N  :  x[c2j.ineinberOf(k)}  k; 

is  tryall<m>(n  in  searcliNodcsi  :  xjcl].ineinberOf(n))  by  (h[n,  kj) 

14  cp.post(x{a]  =-  n); 

is  onFailure 

io  cp.post(x[a]  !=  n); 

,7  } 

«-  } 


Figure  II:  The  Search  Procedure  with  Value  Symmetry  Breaking 


nient  of  the  latter  system  (Ranibo)  will  be  the  basis  for  prototype  implementation  and  eventual 
production-grade  compiler  for  Tempo. 

As  a  part,  of  this  effort,  we  have  addressed  the  problem  of  mapping  Tempo-specified  channels 
used  in  dynamic  distributed  systems  to  executable  code  en  route  to  prototyping  automated  code 
generation. 

Abstract  models  and  specifications  can  be  used  in  the  design  of  cBstributed  applications  to 
formally  reason  about  tlicir  safety  properties.  However,  the  benefits  of  nsing  formal  methods  arc 
often  negated  by  the  ad  hoe  process  of  mapping  the  functionality  of  an  abstract  specification  to 
the  low-level  executable  code  for  target  distributed  platforms.  We  have  developed  the  first  formal 
specification  of  ail  abstract  asynchronous  communication  channel  with  support  for  dynamic  creation 
and  tear  down  of  communication  links  between  participating  network  nodes,  and  its  implementation 
using  Java  sockets.  The  specifications  arc  expressed  using  the  Tempo  formalism,  and  it  is  proved 
that  the  resulting  implementation  preserves  the  safety  properties  of  tl  e  abstract  channel.  This 
approach  can  be  used  to  implement  algorithms  for  dynamic  systems,  where  communicating  nodes 
may  join,  leave,  and  experience  arbitrary  delays.  This  directly  benefits  automated  code  generation 
we  are  targetlmg  in  this  project,  and  we  plan  to  include  an  implemenlal  on  of  such  channels  in  the 
Tempo  toolkit  as  a  standard  building  block  for  dynamic  distributed  systems.  Our  results  appear  in 
the  proceedings  of  2008  IEEE  International  Symposiutn  on  Network  Ccmpuliny  and  Applications 
[121- 

8.1  Rationale:  towards  code  generation 

The  increasing  complexity  of  distributed  software  systems  makes  reasonii  g  about  their  behavior  ev¬ 
ermore  challenging.  Abstract  specifications  of  distributed  systems  simplify  formal  reasoning  about 
their  safely  guarantees,  and  several  formal  systems  have  been  used  for  this  purpose.  However,  this 
abstraction  makes  challenging  the  mapping  of  the  high-level  specification  to  the  facilities  available 
in  a  target  programming  language. 
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Translation  of  abstract  specifications  into  executable  code  for  target  environments  is  particularly 
challenging  in  the  case  of  communication  channels.  Distributed  services  are  designed  for  a  specific 
communication  model,  where  the  safety  properties  of  the  communication  links  used  by  service 
directly  impact  the  safety  guarantees  of  the  overall  system.  Common  practice  often  foregoes  the 
rigorous  safety  arguments  about  the  channel  implementation  and  its  interaction  with  the  system 
components.  Hence,  it  is  not  clear  whether  the  resulting  implementation  is  .correct  with  respect  to 
its  high-level  specification. 

The  key  contribution  of  this  work  is  the.  first  specification  of  an  abstract  asynchronous  commu¬ 
nication  channel  with  explicit  support  of  dynamic  creation  and  tear  down  of  communication  links 
between  the  network  nodes,  and  its  implementation  using  Java  sockets  and  TCP.  For  simplicity, 
our  solution  associates  a  unique  socket  with  each  communication  link  between  a  pair  of  nodes, 
and  thus  it  assumes  that  once  a  node  closes  a  connection  with  some  destination,  it  will  not  try 
to  subsequently  reopen  it.  Our  solution  can  be  naturally  extended  to  incorporate  multiple  con¬ 
current,  point-to-point  socket  connections.  We  prove  that  the  implementation  preserves  the  safety 
guarantees  of  its  abstract  specification. 

In  this  work  we  use  the  Input/Output  Automata  model  to  specify  and  reason  about  the  behavior 
of  distributed  algorithms.  A  plethora  of  algorithms  have  been  described  using  this  model.  We  refer 
to  the  language  used  to  describe  systems  in  this  model  as  IOA.  It  is  of  practical  interest  to  be  able 
to  correctly  specify  and  translate  IOA  models  into  executable  code. 

Tauber  [30]  wrote  the  IOA  compiler,  which  uses  a  target  programming  framework  consisting  of 
Java  and  MP1  The  compiler  design  is  proved  correct  to  ensure  that  the  safety  guarantees  of  the 
source  specification  arc  preserved  by  the  resulting  Java/MPI  implementation.  However,  the  choice 
of  MPI  limits  the  domain  of  systems  to  those  that  do  not  encounter  failures  and  arbitrary  message 
delivery  delays,  and  that  do  not  have  nodes  joining  and  leaving  during  execution.  Given  that  our 
approach  allows  failures,  delays,  and  dynamic  node  participation,  another  direct  application  of 
the  work  presented  here  is  an  alternative  method  of  implementing  robust  communication  channels 
using  TCP  and  Java  sockets.  Note  that  both  methods  of  communication,  i.e.,  Java/MPI  and 
Java  sockets/TCP..  may  be  employed  by  a  compiler,  where  the  first  can  be  chosen  for  failnrc-ficc. 
performance-oriented  applications,  whereas  the  second  is  chosen  for  dynamic  applications  using 
asynchronous  channels. 

8.2  Technical  development:  channel  implementation 

We  present  an  asynchronous  communication  channel  that  connects  applications  running  on  any 
number  of  networked  machines.  Each  sender  node  may  create  connections  with  any  number  of 
receiver  nodes,  and  either  the  sender  node  or  the  receiver  node  may  gracefully  close  the  connection. 
Messages  may  be  lost,  delayed,  and  delivered  out  of  order.  The  current  model  supports  only  a 
single  socket  connection  between  any  two  nodes.  Thus,  once  a  connection  between  two  nodes  is 
established  and  subsequently  closed,  it  cannot  be  reopened  (unless  it  can  be  determined  that  the 
socket  can  be  reused).  Allowing  multiple,  possibly  concurrent,  socket  connections  between  two 
nodes  is  a  straightforward  extension  to  this  model. 

Wc  first  defined  an  automaton,  called  AbsCh,  modeling  the  behavior  of  a  many-to-many,  asyn¬ 
chronous  communication  channel  that  allows  nodes  to  spontaneously  connect  and  disconnect.  The 
connections  are  closed  in  a  graceful  way,  ensuring  that  messages  that  are  in-transit  are  delivered 
before  the  connection  is  closed.  The  signature,  state,  and  transitions  of  AbsCh  are  depicted  in 
Figure  12. 
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Signature: 

Input: 

send(rn,t)j),  where  m  e  A/,  i,j  6  / 
receiverListening(j),  where  j  £  / 
senderOpenfi,.;),  where  i,  j  £  / 
rec.*iverStopUstening(j),  where  j  6  / 
receiverClose(r,  j),  where  i,  j  €  / 
senderClosetiJ),  where  i,j  6  I 


Output: 

feeeive(jn,  i,.;),  where  m  £  1/  and  t,j  £  / 
respRcceiverlistenin where  i,  j  €  / 
Internal: 

senderClosingfi.j),  where  t,  j  £  / 
lose(rn),  where  m  £  A/ 


[ate: 

messages  subset  of  A/  X  /  x  /,  iml  ially  0 
listening,  subset  of  I,  initially  0 

status  :  /  x  /  — *  {closed, connecting, connected)  initially  all  closed 
emptying  :  /  x  /  ifooican,  initially  all  false 


Transitions: 
input  send(r»r1ilj) 

Effect: 

if  status(i,j)  closed  A  -i  emptying  (i,  j)  then 
lucssnjcj  «—  messages  U  {(»»,»,  j}} 


input  receiverListening(g') 
Effect  • 

listening  •—  listening  U  {j) 


input  senderOpen(i,j) 

Effect: 

status  (i,j)  *—  connecting 


out  put  receive( rn,  *>  3 ) 

Precondition: 

(m ,  i,  j)  £  messages 
status(i,j)  =  connected 
Eirn-t : 

messages  *—  messag~r\  {(m,i,  j)} 

output  respReeeiverLtsterog(i, j) 
Precondition: 

status (i,  j)  —  connect  >»g 
j  £  listening 
Effect : 

slotus(i,j)  •—  connect+d 


input  receiverStopListening(j>) 

Effect 

tfsteriiut;  < —  listening  \  (j) 

input  receiverClose(»,  j) 

Effect: 

messages  *—  messages  \  {(m,  s,  r)  £  messagcs|s  =  i  A  r  =  j) 
status {i,  j)  *—  closed 


internal  senderClo$ing(i, j 
Precondition: 
emplymg(i,  j) 

V{m,  s,  r)  £  messages  JjtiArji  j 
Effect: 

status  (i,  j)  ♦—  closed 
emptying (t,  j)  •—  fals« 


input  senderCfose(»,  j) 
Effort. 

emptying(\,j)  —  true 


internal  lose(m) 

Precondil  ion: 

£  messages 

Effect : 

messages  «—  mesiajei  \  {  (m,  i,  j) } 


Figure  12:  Signature,  slate,  3nd  transitions  of  the  abslract  many- to- many  automaton,  AusCif. 
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Figure  13:  Node  automaton. 

Next  we  developed  an  automaton,  called  Jvm-TcpCh,  that  models  the  behavior  of  the  Java 
interface  to  a  communication  channel  using  TCP,  We  do  not  model  TCP  itself  or  the  Java  Virtual 
Machine  (JVM)  environment;  instead,  we  concentrate  on  the  high-level  behavior  and  the  specific 
interface  with  sockets  via  the  Java  libraries. 

Following  Tauber’s  approach  [30],  we  then  establish  a  mediation  between  the  sending  appli¬ 
cation,  the  communication  channel,  and  the  destination  application.  The  mediating  automata 
are  mapped  to  the  nodes  of  the  corresponding  application  automata,  as  illustrated  in  Figure  13, 
showing  a  node  automaton  composed  of  an  application  automaton  and  mediator  automata,  where 
the  mediator  automata  interact  with  the  TCP  sockets  through  the  JVM-TCP  channel  interface. 
We  refer  to  the  composition  of  the  Jvm-TcpCh  automaton  with  the  mediating  automata  as  the 
CompCh  automaton. 

The  method  of  forward  simulation  [1G]  is  used  to  prove  our  main  result  that  CompCh  implements 
AbsCh,  hence  preserving  the  properties  of  our  abstract  asynchronous  channel.  The  full  technical 
development  can  be  found  in  the  available  technical  report.  The  main  result  is  formally  stated  as 
follows. 

Theorem  1  The  set  of  traces  of  CompCh  is  a  subset  of  the  set  of  traces  of  AbsCh. 

9  Conclusion 

This  results  documented  in  this  report  were  developed  under  Phase  I  STTR  contract  for  topic 
AF07-T019.  This  project  advanced  the  state  of  the  art  in  formal  modeling  and  engineering  of 
complex  distributed  systems.  The  project  included:  (a)  modeling  language  that  can  be  used  to 
represent  complex  distributed  systems,  theory  and  methodology  providing  mathematical  basis  for 
modeling  systems  and  reasoning  about  their  properties,  (b)  extensible  and  scalable  analysis  tools 
that  can  be  used  to  validate  correctness  and  performance  properties,  and  synthesis  tools  for  produc¬ 
ing  efficient  deployment  schemes  of  the.  software  components  in  target  networks  subject  to  specified 
constraints.  The  project  extended  the  methodology  to  incorporate  additional  means  for  reasoning 
about  probabilistic  and  hybrid  systems.  The  project  extended  an  integrated  development  environ¬ 
ment,  called  Tempo,  for  modeling,  synthesis,  and  analysis  of  distributed  systems,  developed  tools 
for  efficient  deployment  of  the  software  components  in  target  networks,  and  explored  a  methodology 
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for  generating  code. 

Current  work  on  future  extensions  for  the  Tempo  toolset  and  the  ovenll  methodology  is  funded 
by  XSF,  and  includes  work  on  distributed  code  generation  from  Tempo  specifications  and  opti¬ 
mization  of  distributed  system  deployment  in  target  network  platforms. 

Current  releases  of  Tempo  toolset  for  Linux,  Windows,  and  OSX/PPC  platforms  are  available 
at  www.veromodo.coni. 
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